Crs is shwoing offline in partucular node

When i checked crs status it's showing offline in some node.please can any body tell me how to restart it.Here is the output for this.
--> crs_stat -t
Name Type Target State Host
ora....R1.inst application ONLINE ONLINE drumheller
ora....R2.inst application ONLINE ONLINE nitehawk
ora.BPELPR.db application ONLINE ONLINE drumheller
ora....R1.inst application ONLINE ONLINE drumheller
ora....R2.inst application ONLINE ONLINE nitehawk
ora....PR1.srv application OFFLINE OFFLINE
ora....PR2.srv application OFFLINE OFFLINE
ora...._TAF.cs application OFFLINE OFFLINE
ora.SIEPR.db application ONLINE ONLINE nitehawk
ora....SM1.asm application ONLINE ONLINE drumheller
ora....ER.lsnr application ONLINE ONLINE drumheller
ora....ler.gsd application ONLINE ONLINE drumheller
ora....ler.ons application ONLINE ONLINE drumheller
ora....ler.vip application ONLINE ONLINE drumheller
ora....SM2.asm application ONLINE ONLINE nitehawk
ora....WK.lsnr application ONLINE ONLINE nitehawk
ora....awk.gsd application ONLINE ONLINE nitehawk
ora....awk.ons application ONLINE ONLINE nitehawk
ora....awk.vip application ONLINE ONLINE nitehawk

i think that's service on RAC.
it should start after database started, if it's not start... you have to start by "srvctl" command.
Usage: srvctl start service -d <name> [-s "<service_name_list>" [-i <inst_name>]] [-o <start_options>]
Usage: srvctl status service -d <name> [-s "<service_name_list>"] [-f] [-v] [-S <level>]
Usage: srvctl stop service -d <name> [-s "<service_name_list>" [-i <inst_name>]] [-f]Please check service_name before:
$ crs_stat | less
or
$ crs_stat | grep -A 2 -B 3 'TARGET\=OFFLINE'
after you know service_name ,and then start it
$ srvctl start service -d database_name -s service_nameGood Luck

Similar Messages

  • How to start CRS and other services after rebooting nodes.

    Hi,
    i have created two node cluster database.how to start CRS and other service after rebooting nodes?
    Thanks,

    use crsctl command start...
    $ORA_CRS_HOME/bin/crsctl start crsOr check status about CLusterware...
    $ORA_CRS_HOME/bin/crsctl check crs
    CSS appears healthy
    CRS appears healthy
    EVM appears healthyIf not OK... check log at $ORA_CRS_HOME/log/hostname/*
    and check status database , service and etc...
    use crs_stat
    $ORA_CRS_HOME/bin/crs_stat -t
    $ORA_CRS_HOME/bin/crs_stat if you find some service or instance ... not online...
    use srvctl command, help to start
    Get Help...
    $ORA_CRS_HOME/bin/srvctl -h

  • Start clusterware (CRS) without starting instance in a node

    version ==> 11.2.0.1
    os : Solaris 5.10
    Using
    #$GRID_HOME/bin/crsctl start crsI can start CRS in a node. But , i don't want to start the instance in this node. I just want to start the clusterware and ASM instance in this node.
    Any idea how i can do this?
    Didn't see this option in the help menu for crsctl
    # ./crsctl -h
    Usage: crsctl add       - add a resource, type or other entity
           crsctl backup    - back up voting disk for CSS
           crsctl check     - check a service, resource or other entity
           crsctl config    - output autostart configuration
           crsctl debug     - obtain or modify debug state
           crsctl delete    - delete a resource, type or other entity
           crsctl disable   - disable autostart
           crsctl discover  - discover DHCP server
           crsctl enable    - enable autostart
           crsctl get       - get an entity value
           crsctl getperm   - get entity permissions
           crsctl lsmodules - list debug modules
           crsctl modify    - modify a resource, type or other entity
           crsctl query     - query service state
           crsctl pin       - Pin the nodes in the nodelist
           crsctl relocate  - relocate a resource, server or other entity
           crsctl replace   - replaces the location of voting files
           crsctl release   - release a DHCP lease
           crsctl request   - request a DHCP lease
           crsctl setperm   - set entity permissions
           crsctl set       - set an entity value
           crsctl start     - start a resource, server or other entity
           crsctl status    - get status of a resource or other entity
           crsctl stop      - stop a resource, server or other entity
           crsctl unpin     - unpin the nodes in the nodelist
           crsctl unset     - unset a entity value, restoring its default

    Hi,
    you cannot directly give these information during clusterware startup.
    Only option is to disable the ressources, you do not want to start before starting the node.
    Note: You can do this from another node, to disable the startup on the node you want to start:
    In generall you can do this with
    $ srvctl disable
    The correct syntax, depends on the type of management you used.
    Policy Managed databases use:
    srvctl disable database -d <db_unique_name> -n <node_name>
    Administrator managed databases:
    srvctl disable instance -d <db_unique_name> -i "<inst_name_list>"
    Regards
    Sebastian

  • Crs doesn't start on second node

    Guys,
    RAC on 2 nodes
    Release 10.2.0.5.0
    Solaris 10
    There was a problem with the cable that enables connection for the interconnect, but the problem has been solved. One of the nodes was evicted and all resources were move to the other node. Once the problem was solved I tried to start the cluster that was evicted but to no success. when I run crs_stat -t I get the infamous CRS-0184.
    I have checked the ocr and olsnodes; ocr seems to be fine and the second node is recognized as part of the cluster.
    cluvfy comp ocr -n lenin,trotsky -verbose
    Verifying OCR integrity
    Checking OCR integrity...
    Checking the absence of a non-clustered configuration...
    All nodes free of non-clustered, local-only configurations.
    Uniqueness check for OCR device passed.
    Checking the version of OCR...
    OCR of correct Version "2" exists.
    Checking data integrity of OCR...
    Data integrity check for OCR passed.
    OCR integrity check passed.
    Verification of OCR integrity was successful.
    oracle@trotsky > cluvfy comp nodereach -n lenin,trotsky -srcnode trotsky -verbose
    Verifying node reachability
    Checking node reachability...
    Check: Node reachability from node "trotsky"
    Destination Node Reachable?
    lenin yes
    trotsky yes
    Result: Node reachability check passed from node "trotsky".
    I have checked /var/adm/messages and crs and cssd log but I didn't see anything that stands out...
    I have also tried to delete the content of /var/tmp/.oracle and restart crs but again to no success.
    I have read in another thread in this forum that crs problems are either related to the interconnect or ocr/voting disks but as mentioned before they seem to be OK.
    I'm running out of ideas, any suggestions?
    One of the nodes now holds both vip addresses:
    bge0:1: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
    inet 192.168.191.184 netmask ffffff00 broadcast 192.168.191.255
    bge0:2: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
    inet 192.168.191.182 netmask ffffff00 broadcast 192.168.191.255
    Do I need to manually reconfigure the interface do that is then held by the second node?
    Thanks in advance for your help

    Cheers for your input!
    The results on the suggested cluvfy command is: passed on all checks with the exception of the daemon liveness (as expected).
    Excerpts from the different logs:
    alert.log
    2010-11-19 13:12:35.033
    [cssd(4928)]CRS-1605:CSSD voting file is online: /dev/rdsk/c1t500601604BA03AEAd0s5. Details in /u01/crs/10.2.0/crs_1/log/trotsky/cssd/ocssd.log.
    2010-11-19 13:12:35.050
    [cssd(4928)]CRS-1605:CSSD voting file is online: /dev/rdsk/c1t500601604BA03AEAd0s4. Details in /u01/crs/10.2.0/crs_1/log/trotsky/cssd/ocssd.log.
    2010-11-19 13:12:35.062
    [cssd(4928)]CRS-1605:CSSD voting file is online: /dev/rdsk/c1t500601604BA03AEAd0s6. Details in /u01/crs/10.2.0/crs_1/log/trotsky/cssd/ocssd.log.
    cssd.log
    [    CSSD]2010-11-19 13:16:47.059 [21] >WARNING: clssnmLocalJoinEvent: takeover aborted due to ALIVE node on Disk
    [    CSSD]2010-11-19 13:16:47.059 [21] >WARNING: clssnmRcfgMgrThread: not possible to join the cluster. Please reboot the node.
    [    CSSD]2010-11-19 13:16:47.059 [21] >WARNING: clssnmReconfigThread: state(1) clusterState(0) exit
    I have tried rebooting the node but that did not help.
    crsd.log
    2010-11-19 13:53:49.652: [  CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    2010-11-19 13:53:50.889: [ COMMCRS][1802]clsc_connect: (1009ac310) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_trotsky_))
    2010-11-19 13:53:50.889: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
    2010-11-19 13:53:50.890: [  CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    2010-11-19 13:53:51.899: [    CRSD][1][PANIC] CRSD exiting: Could not init the CSS context
    2010-11-19 13:53:51.899: [    CRSD][1] Done.
    Does this help?

  • Failure at final check of Oracle CRS stack.10  on the second node

    Hi,
    I am trying to install Oracle Clusterware 10.2.0.1.0 in VM machines (2 nodes config) in Linux (OEL5) using VMware Server (2.0). Everything went very well one the first node upto running the root.sh. Running root.sh ended with Failure at final check of Oracle CRS stack 10 error.
    RAC1 root.sh output
    [root@rac1 crs]# ./root.sh
    WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
    WARNING: directory '/u01/crs/oracle/product' is not owned by root
    WARNING: directory '/u01/crs/oracle' is not owned by root
    WARNING: directory '/u01/crs' is not owned by root
    WARNING: directory '/u01' is not owned by root
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up NS directories
    Oracle Cluster Registry configuration upgraded successfully
    WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
    WARNING: directory '/u01/crs/oracle/product' is not owned by root
    WARNING: directory '/u01/crs/oracle' is not owned by root
    WARNING: directory '/u01/crs' is not owned by root
    WARNING: directory '/u01' is not owned by root
    assigning default hostname rac1 for node 1.
    assigning default hostname rac2 for node 2.
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: rac1 rac1-priv rac1
    node 2: rac2 rac2-priv rac2
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Now formatting voting device: /dev/raw/raw2
    Format of 1 voting devices complete.
    Startup will be queued to init within 90 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    CSS is active on these nodes.
    rac1
    CSS is inactive on these nodes.
    rac2
    Local node checking complete.
    Run root.sh on remaining nodes to start CRS daemons.
    [root@rac1 crs]#
    RAC2 root.sh output
    [root@rac2 crs]# ./root.sh
    WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
    WARNING: directory '/u01/crs/oracle/product' is not owned by root
    WARNING: directory '/u01/crs/oracle' is not owned by root
    WARNING: directory '/u01/crs' is not owned by root
    WARNING: directory '/u01' is not owned by root
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up NS directories
    Oracle Cluster Registry configuration upgraded successfully
    WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
    WARNING: directory '/u01/crs/oracle/product' is not owned by root
    WARNING: directory '/u01/crs/oracle' is not owned by root
    WARNING: directory '/u01/crs' is not owned by root
    WARNING: directory '/u01' is not owned by root
    assigning default hostname rac1 for node 1.
    assigning default hostname rac2 for node 2.
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: rac1 rac1-priv rac1
    node 2: rac2 rac2-priv rac2
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Now formatting voting device: /dev/raw/raw2
    Format of 1 voting devices complete.
    Startup will be queued to init within 90 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Failure at final check of Oracle CRS stack.
    10
    [root@rac2 crs]#
    Output of alterrac2.log
    [root@rac2 rac2]# more alertrac2.log
    2009-08-14 23:02:44.699
    [client(5935)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.
    0/crs/log/rac2/client/ocrconfig_5935.log.
    2009-08-14 23:02:44.704
    [client(5935)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.
    0/crs/log/rac2/client/ocrconfig_5935.log.
    2009-08-14 23:02:44.707
    [client(5935)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.
    0/crs/log/rac2/client/ocrconfig_5935.log.
    2009-08-14 23:02:44.864
    [client(5935)]CRS-1001:The OCR was formatted using version 2.
    2009-08-14 23:02:50.339
    [client(6004)]CRS-1801:Cluster crs configured with nodes rac1 rac2 .
    2009-08-14 23:05:07.603
    [cssd(6600)]CRS-1605:CSSD voting file is online: /dev/raw/raw2. Details in /u01/crs/oracle/product/10.2.0/crs/l
    og/rac2/cssd/ocssd.log.
    [root@rac2 rac2]#
    Since raw devices are not supported from OEL5, I did do the workaround in *63-oracle-raw.rules file under /etc/udev/rules.d* dir.
    ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N"
    ACTION=="add", KERNEL=="sdc1", RUN+="/bin/raw /dev/raw/raw2 %N"
    ACTION=="add", KERNEL=="sdd1", RUN+="/bin/raw /dev/raw/raw3 %N"
    ACTION=="add", KERNEL=="sde1", RUN+="/bin/raw /dev/raw/raw4 %N"
    ACTION=="add", KERNEL=="sdf1", RUN+="/bin/raw /dev/raw/raw5 %N"
    KERNEL=="raw[1-2]*", OWNER="root", GROUP="oinstall", MODE="640"
    KERNEL=="raw[3-5]*", OWNER="oracle", GROUP="oinstall", MODE="640"
    One thing I have noticed after running root.sh on both the nodes is the permissons on raw devices changed from
    Before root.sh
    [root@rac2 crs]# ls -ls /dev/raw*
    0 crw------- 1 root root 162, 0 Aug 14 22:42 /dev/rawctl
    /dev/raw:
    total 0
    0 crw-r----- 1 root oinstall 162, 1 Aug 14 22:42 raw1
    0 crw-r----- 1 root oinstall 162, 2 Aug 14 22:42 raw2
    0 crw-r----- 1 oracle oinstall 162, 3 Aug 14 22:42 raw3
    0 crw-r----- 1 oracle oinstall 162, 4 Aug 14 22:42 raw4
    0 crw-r----- 1 oracle oinstall 162, 5 Aug 14 22:42 raw5
    to
    [root@rac2 crs]# ls -ls /dev/raw*
    0 crw------- 1 root root 162, 0 Aug 14 22:31 /dev/rawctl
    /dev/raw:
    total 0
    0 crw-r----- 1 root oinstall 162, 1 Aug 14 22:56 raw1
    0 crw-r--r-- 1 oracle oinstall 162, 2 Aug 14 23:01 raw2
    0 crw-r----- 1 oracle oinstall 162, 3 Aug 14 22:31 raw3
    0 crw-r----- 1 oracle oinstall 162, 4 Aug 14 22:31 raw4
    0 crw-r----- 1 oracle oinstall 162, 5 Aug 14 22:31 raw5
    [root@rac1 crs]#
    My shared disk listing
    [root@www shared]# ls -ltr
    total 8780
    -rw------- 1 root root 640 Aug 14 22:43 votingdisk.vmdk
    -rw------- 1 root root 598 Aug 14 22:43 ocr.vmdk
    -rw------- 1 root root 604 Aug 14 22:43 asm3.vmdk
    -rw------- 1 root root 604 Aug 14 22:43 asm2.vmdk
    -rw------- 1 root root 604 Aug 14 22:43 asm1.vmdk
    -rw------- 1 root root 65536 Aug 14 22:44 votingdisk-s006.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s005.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s004.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s003.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s002.vmdk
    -rw------- 1 root root 393216 Aug 14 22:44 votingdisk-s001.vmdk
    -rw------- 1 root root 65536 Aug 14 22:44 ocr-s006.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 ocr-s005.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 ocr-s004.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 ocr-s003.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 ocr-s002.vmdk
    -rw------- 1 root root 393216 Aug 14 22:44 ocr-s001.vmdk
    -rw------- 1 root root 65536 Aug 14 22:44 asm3-s006.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm3-s005.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm3-s004.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm3-s003.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm3-s002.vmdk
    -rw------- 1 root root 393216 Aug 14 22:44 asm3-s001.vmdk
    -rw------- 1 root root 65536 Aug 14 22:44 asm2-s006.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm2-s005.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm2-s004.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm2-s003.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm2-s002.vmdk
    -rw------- 1 root root 393216 Aug 14 22:44 asm2-s001.vmdk
    -rw------- 1 root root 65536 Aug 14 22:44 asm1-s006.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm1-s005.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm1-s004.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm1-s003.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm1-s002.vmdk
    -rw------- 1 root root 393216 Aug 14 22:44 asm1-s001.vmdk
    [root@www shared]#
    I don't know how to fix this problem. I did go through many docs and metalink notes.
    I am new to RAC world. It took 3 days to come to this stage. Please help me.
    Thanks
    Leo

    Hi Surachart,
    Here is my messages output..
    */var/log/messages*
    Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for fe80::20c:29ff:fe6b:f9a8 on eth1.
    Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for 192.168.1.196 on eth1.
    Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for fe80::20c:29ff:fe6b:f99e on eth0.
    Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for 192.168.0.196 on eth0.
    Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering HINFO record with values 'I686'/'LINUX'.
    Aug 20 14:05:02 rac2 avahi-daemon[3627]: Server startup complete. Host name is rac2.local. Local service cookie is 927471131.
    Aug 20 14:05:03 rac2 avahi-daemon[3627]: Service "SFTP File Transfer on rac2" (/services/sftp-ssh.service) successfully established.
    Aug 20 14:05:08 rac2 smartd[3739]: smartd version 5.38 [i686-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
    Aug 20 14:05:08 rac2 smartd[3739]: Home page is http://smartmontools.sourceforge.net/
    Aug 20 14:05:08 rac2 smartd[3739]: Opened configuration file /etc/smartd.conf
    Aug 20 14:05:08 rac2 smartd[3739]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/hdc, opened
    Aug 20 14:05:08 rac2 kernel: hdc: drive_cmd: status=0x51 { DriveReady SeekComplete Error }
    Aug 20 14:05:08 rac2 kernel: hdc: drive_cmd: error=0x04 { AbortedCommand }
    Aug 20 14:05:08 rac2 kernel: ide: failed opcode was: 0xec
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/hdc, not ATA, no IDENTIFY DEVICE Structure
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sda, opened
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sda, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on SMART features
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdb, opened
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdb, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdb' to turn on SMART features
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdc, opened
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdc, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdc' to turn on SMART features
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdd, opened
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdd, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdd' to turn on SMART features
    Aug 20 14:05:09 rac2 smartd[3739]: Device: /dev/sde, opened
    Aug 20 14:05:09 rac2 smartd[3739]: Device: /dev/sde, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sde' to turn on SMART features
    Aug 20 14:05:10 rac2 smartd[3739]: Device: /dev/sdf, opened
    Aug 20 14:05:10 rac2 smartd[3739]: Device: /dev/sdf, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdf' to turn on SMART features
    Aug 20 14:05:10 rac2 smartd[3739]: Monitoring 0 ATA and 0 SCSI devices
    Aug 20 14:05:10 rac2 smartd[3741]: smartd has fork()ed into background mode. New PID=3741.
    Aug 20 14:05:13 rac2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
    Aug 20 14:05:13 rac2 last message repeated 3 times
    Aug 20 14:05:27 rac2 gconfd (root-3967): starting (version 2.14.0), pid 3967 user 'root'
    Aug 20 14:05:27 rac2 gconfd (root-3967): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
    Aug 20 14:05:27 rac2 gconfd (root-3967): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 1
    Aug 20 14:05:27 rac2 gconfd (root-3967): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
    Aug 20 14:05:29 rac2 gconfd (root-3967): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 0
    Aug 20 14:05:29 rac2 hald: mounted /dev/hdc on behalf of uid 0
    Aug 20 14:05:29 rac2 hcid[3311]: Default passkey agent (:1.8, /org/bluez/applet) registered
    Aug 20 14:05:31 rac2 nm-system-settings: Loaded plugin ifcfg-rh: (c) 2007 - 2008 Red Hat, Inc. To report bugs please use the NetworkManager mailing list.
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth1 ...
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: read connection 'System eth1'
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-lo ...
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: error: Ignoring loopback device config.
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth0 ...
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: read connection 'System eth0'
    Aug 20 14:05:31 rac2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
    Aug 20 14:05:32 rac2 last message repeated 4 times
    Aug 20 14:12:51 rac2 kernel: FS-Cache: Loaded
    Aug 20 14:22:06 rac2 xinetd[3488]: START: shell pid=5193 from=192.168.0.195
    Aug 20 14:22:06 rac2 xinetd[3488]: EXIT: shell status=0 pid=5193 duration=0(sec)
    Aug 20 14:22:07 rac2 xinetd[3488]: START: shell pid=5217 from=192.168.0.195
    Aug 20 14:22:07 rac2 xinetd[3488]: EXIT: shell status=0 pid=5217 duration=0(sec)
    Aug 20 14:22:07 rac2 xinetd[3488]: START: shell pid=5241 from=192.168.0.195
    Aug 20 14:22:07 rac2 xinetd[3488]: EXIT: shell status=0 pid=5241 duration=0(sec)
    Aug 20 14:22:16 rac2 xinetd[3488]: EXIT: shell status=0 pid=6236 duration=0(sec)
    Aug 20 14:22:16 rac2 xinetd[3488]: START: shell pid=6265 from=192.168.0.195
    Aug 20 14:22:16 rac2 xinetd[3488]: EXIT: shell status=0 pid=6265 duration=0(sec)
    Aug 20 14:22:16 rac2 xinetd[3488]: START: shell pid=6291 from=192.168.0.195
    Aug 20 14:22:17 rac2 xinetd[3488]: EXIT: shell status=0 pid=6291 duration=1(sec)
    Aug 20 14:22:17 rac2 xinetd[3488]: START: shell pid=6317 from=192.168.0.195
    Aug 20 14:22:17 rac2 xinetd[3488]: EXIT: shell status=0 pid=6317 duration=0(sec)
    [root@rac2 log]#

  • Timed out waiting for the CRS stack to start on Second node.

    Hi,
    I am trying to setup 2 node 11gR2 RAC in vmware. I face the Timed out waiting for the CRS stack to start error on second node while running root.sh.I checked the cluster log files located in  /u01/app/11.2.0/grid/log/node1/alertnode2.log it shows as mentioned below.But when i logged into ASM instance and checked the diskgroup and it is in mount state on both nodes,So i am really confused why i ended up with error in 2nd node on running root.sh. Can anyone tell me how to correct this error or any other checking needs to be done?
    /u01/app/11.2.0/grid/log/node1/alertnode2.log
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:10.314
    [crsd(25309)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
    2014-01-21 21:40:11.217
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:12.422
    [crsd(25335)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
    2014-01-21 21:40:13.327
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:14.514
    [crsd(25356)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
    2014-01-21 21:40:15.439
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:15.440
    [ohasd(23261)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    root.sh output
    [root@node2 ~]# /u01/app/11.2.0/grid/root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /u01/app/11.2.0/grid
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
    The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
    [n]: y
       Copying dbhome to /usr/local/bin ...
    The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
    [n]: y
       Copying oraenv to /usr/local/bin ...
    The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
    [n]: y
       Copying coraenv to /usr/local/bin ...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2014-01-21 21:37:55: Parsing the host name
    2014-01-21 21:37:55: Checking for super user privileges
    2014-01-21 21:37:55: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_p                   
    arams
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node 11gdb, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'node2'
    CRS-2676: Start of 'ora.mdnsd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'node2'
    CRS-2676: Start of 'ora.gipcd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'node2'
    CRS-2676: Start of 'ora.gpnpd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'node2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'node2'
    CRS-2676: Start of 'ora.diskmon' on 'node2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'node2'
    CRS-2676: Start of 'ora.ctssd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'node2'
    CRS-2676: Start of 'ora.drivers.acfs' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'node2'
    CRS-2676: Start of 'ora.asm' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.crsd' on 'node2'
    CRS-2676: Start of 'ora.crsd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 'node2'
    CRS-2676: Start of 'ora.evmd' on 'node2' succeeded
    Timed out waiting for the CRS stack to start.
    Regards,
    007

    Hi,
    Are you trying to install on Vmware Workstation or Vmware Server?
    If you are using vmware server add below line in your .vmx file for both node
    scsi1.sharedBus = "VIRTUAL"
    First check which scsi serial number are you using as above I have selected scsi1 serial during disk addition, 

  • CRS oui failed to find the node

    Hi every friend,
    I am trying to install the CRS on 2 solaris 9 (VERITAS CFS). But when the oui came to the page "Specify Cluster Configuration" page, The OUI can not find any node and also the ADD, EDIT , REMOVE three buttons unpressable with grey color. From the help I know that when OUI detect the third-party vendor clusterware, It will use the node infomation obtained from third-party vendor clusterware and these infomation is not able to be edited by OUI. But it seems that it can not obtain anynode infomation. I had ping the PUB/PRIV IP of both node and they are accessible. How can I make the OUI to find the node infomation?
    Thank you very much!

    Hi,
    <p>
    Yes, it is. <br>
    Environment >> Servers >> jmsserver2 >> Configuration >> General > Machine : sv-jms2
    </p>
    regards
    <br>Lukas<br>
    <br>WLS 9.2

  • CRS - Making Registry changes to local node while CRS running

    Hello All,
    Just a curiosity question. I've made created some bad setting in my OCR/OLR. Now I cannot startup CRS.
    Is it possible to somewhere manualy edit a certain resource (say network) ?
    Oh, and I dod not have a backup of the OLR just a very old one which is useless. I however do have 1 node still running.
    Thanks
    Jan S.

    Please try command crsctl modify res and see whether it gives any error or not. What settings you want to change?

  • TAF Service OFFLINE on standby node

    Hi,
    We have created primary and secondary RAC. A TAF service which is ONLINE on primary is added to standby side and is enabled but it shows the OFFLINE status. Is this default behaviour. We can't start is using "srvctl start service" command. My understanding is since the DB is physical standby which is not open, this is the reason the service is OFF line. Once the standby switches to primary it can be brought up. Correct?

    You should still be able to bring up the standby as read only. During managed recovery the database is in a mounted state and therefore can't be connected to via the listener.

  • One node is up but when we start the crs on node 2 then ASM instance killed.

    Hi Friends
    i am facing below strange problem
    one node is up but when we start the crs on node 2 then ASM instance killed. i found that one ip 169.254.98.19 is binding with private interconnect IP rathter than assigned IP.
    Please help me

    Hi Pradeep ,
    Oracle S/W verstion is 11.2.0.3. Actually , we start the CRS and DB on node 1 but when we try to start on node 2 , getting below error.
    [gpnpd(18284748)]CRS-2328:GPNPD started on node ggnetp04.
    2013-09-03 19:48:29.162
    [cssd(17694820)]CRS-1713:CSSD daemon is started in clustered mode
    2013-09-03 19:48:30.557
    [ohasd(13697160)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
    2013-09-03 19:48:41.013
    [cssd(17694820)]CRS-1707:Lease acquisition for node ggnetp04 number 2 completed
    2013-09-03 19:48:42.490
    [cssd(17694820)]CRS-1605:CSSD voting file is online: /dev/rhdiskpower0; details in /opt/app/11.2.0.3/grid/log/ggnetp04/cssd/ocssd.log.
    2013-09-03 19:48:45.483
    [cssd(17694820)]CRS-1601:CSSD Reconfiguration complete. Active nodes are ggnetp03 ggnetp04 .
    2013-09-03 19:48:47.257
    [ctssd(18022612)]CRS-2403:The Cluster Time Synchronization Service on host ggnetp04 is in observer mode.
    2013-09-03 19:48:47.632
    [ctssd(18022612)]CRS-2407:The new Cluster Time Synchronization Service reference node is host ggnetp03.
    2013-09-03 19:48:47.633
    [ctssd(18022612)]CRS-2401:The Cluster Time Synchronization Service started on host ggnetp04.
    [client(16056466)]CRS-10001:03-Sep-13 19:48 ACFS-9391: Checking for existing ADVM/ACFS installation.
    [client(16056468)]CRS-10001:03-Sep-13 19:48 ACFS-9392: Validating ADVM/ACFS installation files for operating system.
    [client(16056470)]CRS-10001:03-Sep-13 19:48 ACFS-9393: Verifying ASM Administrator setup.
    [client(16056472)]CRS-10001:03-Sep-13 19:48 ACFS-9308: Loading installed ADVM/ACFS drivers.
    [client(16056478)]CRS-10001:03-Sep-13 19:48 ACFS-9154: Loading 'oracleadvm.ext' driver.
    [client(16056486)]CRS-10001:03-Sep-13 19:48 ACFS-9154: Loading 'oracleacfs.ext' driver.
    [client(16056494)]CRS-10001:03-Sep-13 19:48 ACFS-9327: Verifying ADVM/ACFS devices.
    [client(16056498)]CRS-10001:03-Sep-13 19:48 ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
    [client(16056504)]CRS-10001:03-Sep-13 19:48 ACFS-9156: Detecting control device '/dev/ofsctl'.
    [client(16056508)]CRS-10001:03-Sep-13 19:48 ACFS-9322: completed
    2013-09-03 19:48:59.667
    [ctssd(18022612)]CRS-2409:The clock on host ggnetp04 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2013-09-03 19:50:53.936
    [cssd(17694820)]CRS-1662:Member kill requested by node ggnetp03 for member number 1, group DB+ASM
    2013-09-03 19:50:57.493
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:50:57.494
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-09-03 19:50:58.828
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:50:58.943
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:50:59.235
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:53:00.252
    [cssd(17694820)]CRS-1662:Member kill requested by node ggnetp03 for member number 1, group DB+ASM
    2013-09-03 19:53:03.101
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:53:03.102
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-09-03 19:53:05.430
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:53:05.539
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:53:05.815
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:55:07.436
    [cssd(17694820)]CRS-1662:Member kill requested by node ggnetp03 for member number 1, group DB+ASM
    2013-09-03 19:55:09.673
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:55:09.674
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-09-03 19:55:12.007
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:55:12.115
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:55:12.389
    [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".
    2013-09-03 19:55:12.405
    [ohasd(13697160)]CRS-2807:Resource 'ora.asm' failed to start automatically.
    2013-09-03 19:55:12.405
    [ohasd(13697160)]CRS-2807:Resource 'ora.crsd' failed to start automatically.
    2013-09-03 20:23:32.040
    [ctssd(18022612)]CRS-2409:The clock on host ggnetp04 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

  • Error when raise crs on one node

    rac with three-node，linux redhat as 5+10.2.0.4
    on rac01：
    [root@rac01 bin]# *./crsctl stop crs*
    Stopping resources. This could take several minutes.
    Error while stopping resources. Possible cause: CRSD is down.
    [root@rac01 bin]# *./crsctl check css*
    Failure 1 contacting CSS daemon
    [root@rac01 bin]# *./crsctl check crs*
    Failure 1 contacting CSS daemon
    Cannot communicate with CRS
    Cannot communicate with EVM
    [root@rac01 bin]# *./crsctl start crs*
    Attempting to start CRS stack
    The CRS stack will be started shortly
    [root@rac01 bin]# *./crs_stat -t*
    Message not found
    [root@rac01 bin]# ps -ef | grep oracle | grep "\.bin" | grep -v grep
    oracle 3815 3814 1 23:26 ? 00:00:05 /home/oracle/product/10.2.0/crs/bin/evmd.bin
    oracle 5160 5132 1 23:28 ? 00:00:02 /home/oracle/product/10.2.0/crs/bin/oclsomon.bin
    root 6991 6569 0 22:29 ? 00:00:00 /home/oracle/product/10.2.0/crs/bin/oprocd.bin run -t 1000 -m 500 -f
    oracle 7169 6600 0 22:29 ? 00:00:27 /home/oracle/product/10.2.0/crs/bin/ocssd.bin
    root 30004 29673 1 23:14 ? 00:00:18 /home/oracle/product/10.2.0/crs/bin/crsd.bin restart
    log outputs：
    [root@rac01 cssd]# tail -100f ocssd.log
    [    CSSD]2011-06-14 00:03:44.686 [1209174336] >TRACE: Authentication OSD error, op: scls_auth_response_prepare
    loc: validate4
    info: directory for /home/oracle/product/10.2.0/crs/css/auth/A2513039 does not exist
    dep: 2
    [    CSSD]2011-06-14 00:03:45.001 [1209174336] >TRACE: Authentication OSD error, op: scls_auth_response_prepare
    loc: validate4
    info: directory for /home/oracle/product/10.2.0/crs/css/auth/A4520272 does not exist
    dep: 2
    [    CSSD]2011-06-14 00:03:45.011 [1209174336] >TRACE: Authentication OSD error, op: scls_auth_response_prepare
    loc: validate4
    info: directory for /home/oracle/product/10.2.0/crs/css/auth/A1135239 does not exist
    dep: 2
    [    CSSD]2011-06-14 00:03:46.001 [1209174336] >TRACE: Authentication OSD error, op: scls_auth_response_prepare
    loc: validate4
    info: directory for /home/oracle/product/10.2.0/crs/css/auth/A3144859 does not exist
    [oracle@rac01 crsd]$ tail -100f crsd.log
    2011-06-14 00:04:01.284: [ COMMCRS][2606041840]clsc_auth_send: (0x12abec80) Connection not active
    2011-06-14 00:04:01.284: [ COMMCRS][2606041840]Authorization failed, network error
    2011-06-14 00:04:01.284: [ CSSCLNT][2606041840]clsssInitNative: connect failed, rc 2
    2011-06-14 00:04:01.286: [  CRSRTI][2606041840]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    [  OCRUTL][2606041840]u_set_comp_error: Parameter was NULL
    [  OCRUTL][2606041840]u_set_ocr_error: Parameter was NULL
    2011-06-14 00:04:02.681: [ COMMCRS][2606041840]Authorization failed, network error
    2011-06-14 00:04:02.681: [ CSSCLNT][2606041840]clsssInitNative: connect failed, rc 2
    [oracle@rac01 evmd]$ tail -10f evmd.log
    2011-06-13 23:45:29.059: [ CSSCLNT][319068720]clsssInitNative: connect failed, rc 2
    2011-06-13 23:45:29.059: [    EVMD][319068720]0EVMD waiting for CSS to be ready err = 3
    [  OCRUTL][319068720]u_set_comp_error: Parameter was NULL
    [  OCRUTL][319068720]u_set_ocr_error: Parameter was NULL
    2011-06-13 23:45:30.345: [ COMMCRS][319068720]Authorization failed, network error
    2011-06-13 23:45:30.345: [ CSSCLNT][319068720]clsssInitNative: connect failed, rc 2
    2011-06-13 23:45:30.346: [    EVMD][319068720]0EVMD waiting for CSS to be ready err = 3
    [  OCRUTL][319068720]u_set_comp_error: Parameter was NULL
    [  OCRUTL][319068720]u_set_ocr_error: Parameter was NULL
    processes on rac02：
    [oracle@rac02 ~]$ ps -ef | grep oracle | grep "\.bin" | grep -v grep
    oracle 6334 6330 0 16:53 ? 00:00:02 /home/oracle/product/10.2.0/crs/bin/evmd.bin
    root 6464 5524 0 16:53 ? 00:00:27 /home/oracle/product/10.2.0/crs/bin/crsd.bin reboot
    root 6898 6475 0 16:53 ? 00:00:00 /home/oracle/product/10.2.0/crs/bin/oprocd.bin run -t 1000 -m 500 -f
    oracle 7007 6954 0 16:53 ? 00:00:00 /home/oracle/product/10.2.0/crs/bin/oclsomon.bin
    oracle 7069 6509 0 16:53 ? 00:00:17 /home/oracle/product/10.2.0/crs/bin/ocssd.bin
    oracle    7218  6334  0 16:54 ?        00:00:00 /home/oracle/product/10.2.0/crs/bin/evmlogger.bin -o /home/oracle/product/10.2.0/crs/evm/log/evmlogger.info -l /home/oracle/product/10.2.0/crs/evm/log/evmlogger.log
    why node rac01's crs service not raise ?

    Hi;
    For your issue i suggest close your thread here as changing thread status to answered and move it to Forum Home » Grid Computing » Real Application Clusters which you can get more quick response
    Regard
    Helios

  • VIP in 11g RAC with CRS

    Hi all,
    My system is a three nodes 11.1.0.7 RAC cluster with CRS (same version).
    I have a question about how VIP works.
    Notes:
    eth0 - public interface
    eth0:1 - vip interface
    eth1: private interface
    If I go down with eth0 (ifconfig eth0 down), VIP is relocated to other of the two remain network living nodes. Everything is OK at this moment and only listener resource in CRS stack is OFFLINE, even INST, GSD and ONS are ONLINE and remains in the first node.
    Then I go up with the public interface (ifconfig eth0 up). Now everything still remains in the same state... Nothing Change!!!
    Questions:
    - Node1 VIP resource should go back to the first node???
    - Listener resource should go online again???
    I can manually start listener resource or relocate vip to the node1 without a problem... but it should be done without manual word, isn't it????
    Thanks in advance.

    user605547 wrote:
    I know that listener is nodeapps (vip) dependent... so if VIP is not running up, listener can't start
    All my issue is about automatic VIP relocating after a network failure, so I'm trying to understad what happens and what it should happens.
    Scenario 1:
    Step 1:
    NODE1: ifconfig eth0 down => so vip fails over node2 and listener on node1 also goes down. Any other resource in node1 remains ONLINE.
    Step 2:
    NODE1: ifconfig eth0 up => nothing happens and everything remains in same state. NODE1 VIP is still running in node2
    Question: NODE1 VIP should automatic relocate to node 1 (from node2)??????
    11gR1, NODE1 VIP not automatic to node01, we still relocate manual.
    This is actually the default default behavior in 10.2.0.4 and 11.1.
    Step 3:
    NODE1: srvctl start listener -n node1 -l listener_node1 => node1 vip relocates to node 1 and listener_node1 goes upnot sure, perhaps srvctl check about public, then relocate VIP
    Scenario 2:
    Step 1:
    NODE1: ifconfig eth0 down => so vip fails over node2 and listener on node1 also goes down. Any other resource in node1 remains ONLINE.
    Step 2:
    NODE1: ifconfig eth0 up => nothing happens and everything remains in same state. NODE1 VIP is still running in node2
    Step 3:
    NODE3: srvctl stop listener -n node3 -l listener_node3 (changing state of any resource not involved with node1) =>
    => listener in node3 goes down, node1_vip relocates from node2 to node1 and listener in node1 goes up
    Is it normal??????? It looks like any operation in CRS forces to check NODE1_VIP availability and make it goes up in original node??? Expected behaviour???
    When we use srvctl, instance check is performed, that check public ip -> resolved and then relocate VIP
    How about ?
    ORA_RACG_VIP_FAILBACK in the racgwrap script in the ORACLE_HOME

  • CRS on Redhat 5 auto-start problem

    Dear all
    Recently I tried to install Oracle 10gR2 on Redhat 5. I did research and took all the necessary actions to install it on Redhat. Everything went well, according to the documents and at the end of the installation, all the resources were up and running. However, after the install, only the VIP was up and ONS and GSD was in unknown format. When I stop the nodeapps and start it again, everything stars working. It seems that the problem is something to do with boot process. I tried to put some delay ( for OCR mount ) but still no luck. This is the result from crsd.log from node1 when node2 is down.. :
    2009-06-24 22:00:07.461: [ default][3594852768]0CRS Daemon Starting
    2009-06-24 22:00:07.461: [ CRSMAIN][3594852768]0Checking the OCR device
    2009-06-24 22:00:07.464: [ CRSMAIN][3594852768]0Connecting to the CSS Daemon
    2009-06-24 22:00:07.838: [ COMMCRS][1103116608]clsc_connect: (0x151f20d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1dr_crsdr))
    2009-06-24 22:00:07.838: [ CSSCLNT][3594852768]clsssInitNative: connect failed, rc 9
    2009-06-24 22:00:07.838: [  CRSRTI][3594852768]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    2009-06-24 22:00:09.307: [ COMMCRS][1103116608]clsc_connect: (0x151f23b0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1dr_crsdr))
    2009-06-24 22:00:09.307: [ CSSCLNT][3594852768]clsssInitNative: connect failed, rc 9
    2009-06-24 22:00:09.307: [  CRSRTI][3594852768]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    2009-06-24 22:00:10.773: [ COMMCRS][1103116608]clsc_connect: (0x151f25c0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1dr_crsdr))
    2009-06-24 22:00:10.773: [ CSSCLNT][3594852768]clsssInitNative: connect failed, rc 9
    2009-06-24 22:00:10.774: [  CRSRTI][3594852768]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    2009-06-24 22:00:12.150: [ COMMCRS][1103116608]clsc_connect: (0x151f2880) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1dr_crsdr))
    2009-06-24 22:00:12.150: [ CSSCLNT][3594852768]clsssInitNative: connect failed, rc 9
    2009-06-24 22:00:12.150: [  CRSRTI][3594852768]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    2009-06-24 22:00:19.093: [    CRSD][3594852768]0Daemon Version: 10.2.0.1.0 Active Version: 10.2.0.1.0
    2009-06-24 22:00:19.093: [    CRSD][3594852768]0Active Version and Software Version are same
    2009-06-24 22:00:19.093: [ CRSMAIN][3594852768]0Initializing OCR
    2009-06-24 22:00:19.098: [  OCRRAW][3594852768]proprioo: for disk 0 (/u02/oradata/orcl/OCRFile), id match (1), my id set (503252545,1028247821) total id sets (1), 1st set (503252545,1028247821), 2nd set (0,0) my votes (2), total votes (2)
    2009-06-24 22:00:19.128: [  OCRMAS][1228994880]th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number = 1
    2009-06-24 22:00:19.129: [  OCRRAW][1228994880]proprioo: for disk 0 (/u02/oradata/orcl/OCRFile), id match (1), my id set (503252545,1028247821) total id sets (1), 1st set (503252545,1028247821), 2nd set (0,0) my votes (2), total votes (2)
    2009-06-24 22:00:19.144: [    CRSD][3594852768]0ENV Logging level for Module: allcomp 0
    2009-06-24 22:00:19.144: [    CRSD][3594852768]0ENV Logging level for Module: default 0
    2009-06-24 22:00:19.145: [    CRSD][3594852768]0ENV Logging level for Module: COMMCRS 0
    2009-06-24 22:00:19.145: [    CRSD][3594852768]0ENV Logging level for Module: COMMNS 0
    2009-06-24 22:00:19.146: [    CRSD][3594852768]0ENV Logging level for Module: CRSUI 0
    2009-06-24 22:00:19.146: [    CRSD][3594852768]0ENV Logging level for Module: CRSCOMM 0
    2009-06-24 22:00:19.148: [    CRSD][3594852768]0ENV Logging level for Module: CRSRTI 0
    2009-06-24 22:00:19.148: [    CRSD][3594852768]0ENV Logging level for Module: CRSMAIN 0
    2009-06-24 22:00:19.149: [  OCRSRV][1249974592]th_select_handler: Failed to retrieve procctx from ht. constr = [363817168] retval lht [-27] Signal CV.
    2009-06-24 22:00:19.149: [    CRSD][3594852768]0ENV Logging level for Module: CRSPLACE 0
    2009-06-24 22:00:19.149: [    CRSD][3594852768]0ENV Logging level for Module: CRSAPP 0
    2009-06-24 22:00:19.150: [    CRSD][3594852768]0ENV Logging level for Module: CRSRES 0
    2009-06-24 22:00:19.150: [    CRSD][3594852768]0ENV Logging level for Module: CRSOCR 0
    2009-06-24 22:00:19.151: [    CRSD][3594852768]0ENV Logging level for Module: CRSTIMER 0
    2009-06-24 22:00:19.151: [    CRSD][3594852768]0ENV Logging level for Module: CRSEVT 0
    2009-06-24 22:00:19.152: [    CRSD][3594852768]0ENV Logging level for Module: CRSD 0
    2009-06-24 22:00:19.153: [    CRSD][3594852768]0ENV Logging level for Module: CLUCLS 0
    2009-06-24 22:00:19.154: [    CRSD][3594852768]0ENV Logging level for Module: OCRRAW 0
    2009-06-24 22:00:19.155: [    CRSD][3594852768]0ENV Logging level for Module: OCROSD 0
    2009-06-24 22:00:19.155: [    CRSD][3594852768]0ENV Logging level for Module: CSSCLNT 0
    2009-06-24 22:00:19.156: [    CRSD][3594852768]0ENV Logging level for Module: OCRAPI 0
    2009-06-24 22:00:19.157: [    CRSD][3594852768]0ENV Logging level for Module: OCRUTL 0
    2009-06-24 22:00:19.158: [    CRSD][3594852768]0ENV Logging level for Module: OCRMSG 0
    2009-06-24 22:00:19.159: [    CRSD][3594852768]0ENV Logging level for Module: OCRCLI 0
    2009-06-24 22:00:19.159: [    CRSD][3594852768]0ENV Logging level for Module: OCRCAC 0
    2009-06-24 22:00:19.160: [    CRSD][3594852768]0ENV Logging level for Module: OCRSRV 0
    2009-06-24 22:00:19.160: [    CRSD][3594852768]0ENV Logging level for Module: OCRMAS 0
    2009-06-24 22:00:19.161: [ CRSMAIN][3594852768]0Filename is /u01/app/crs/crs/init/rac1dr.pid
    [  clsdmt][1417812288]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=rac1drDBG_CRSD))
    2009-06-24 22:00:19.506: [ CRSMAIN][3594852768]0Using Authorizer location: /u01/app/crs/crs/auth/
    2009-06-24 22:00:19.760: [ CRSMAIN][3594852768]0Initializing RTI
    2009-06-24 22:00:19.761: [CRSTIMER][1438792000]0Timer Thread Starting.
    2009-06-24 22:00:19.770: [  CRSRES][3594852768]0Parameter SECURITY = 1, running in USER Mode
    2009-06-24 22:00:19.770: [ CRSMAIN][3594852768]0Initializing EVMMgr
    2009-06-24 22:00:19.996: [ COMMCRS][1449281856]clsc_connect: (0x15b1dc20) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth))
    2009-06-24 22:00:21.860: [ CRSMAIN][3594852768]0CRSD locked during state recovery, please wait.
    2009-06-24 22:00:22.494: [ CRSMAIN][3594852768]0CRSD recovered, unlocked.
    2009-06-24 22:00:22.496: [ CRSMAIN][3594852768]0QS socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))
    2009-06-24 22:00:22.604: [ CRSMAIN][3594852768]0CRSD UI socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET))
    2009-06-24 22:00:22.607: [ CRSMAIN][3594852768]0E2E socket on: (ADDRESS=(PROTOCOL=tcp)(HOST=rac1dr-priv)(PORT=49896))
    2009-06-24 22:00:22.607: [ CRSMAIN][3594852768]0Starting Threads
    2009-06-24 22:00:22.607: [ CRSMAIN][3594852768]0CRS Daemon Started.
    2009-06-24 22:00:23.732: [  CRSRES][1554180416]0startRunnable: setting CLI values
    2009-06-24 22:00:23.821: [  CRSRES][1554180416]0Attempting to start `ora.rac2dr.vip` on member `rac1dr`
    2009-06-24 22:00:23.822: [  CRSRES][1543690560]0startRunnable: setting CLI values
    2009-06-24 22:00:23.935: [  CRSRES][1543690560]0Attempting to start `ora.rac1dr.vip` on member `rac1dr`
    2009-06-24 22:00:30.657: [  CRSRES][1554180416]0Start of `ora.rac2dr.vip` on member `rac1dr` succeeded.
    2009-06-24 22:00:30.705: [  CRSRES][1543690560]0Start of `ora.rac1dr.vip` on member `rac1dr` succeeded.
    2009-06-24 22:00:31.408: [  CRSRES][1554180416]0startRunnable: setting CLI values
    2009-06-24 22:00:31.410: [  CRSRES][1543690560]0startRunnable: setting CLI values
    2009-06-24 22:00:31.647: [  CRSRES][1554180416]0Attempting to start `ora.rac1dr.gsd` on member `rac1dr`
    2009-06-24 22:00:31.708: [  CRSRES][1543690560]0Attempting to start `ora.rac1dr.ons` on member `rac1dr`
    2009-06-24 22:00:32.538: [  CRSEVT][1554180416]0CAAMonitorHandler :: 0:Action Script /u01/app/crs/bin/racgwrap(start) timed out for ora.rac1dr.gsd! (timeout=600)
    2009-06-24 22:00:32.538: [  CRSAPP][1554180416]0StartResource error for ora.rac1dr.gsd error code = -2
    2009-06-24 22:00:32.573: [  CRSEVT][1543690560]0CAAMonitorHandler :: 0:Action Script /u01/app/crs/bin/racgwrap(start) timed out for ora.rac1dr.ons! (timeout=600)
    2009-06-24 22:00:32.573: [  CRSAPP][1543690560]0StartResource error for ora.rac1dr.ons error code = -2
    2009-06-24 22:00:32.737: [  CRSEVT][1554180416]0CAAMonitorHandler :: 0:Action Script /u01/app/crs/bin/racgwrap(stop) timed out for ora.rac1dr.gsd! (timeout=600)
    2009-06-24 22:00:32.737: [  CRSAPP][1554180416]0StopResource error for ora.rac1dr.gsd error code = -2
    2009-06-24 22:00:32.837: [  CRSEVT][1543690560]0CAAMonitorHandler :: 0:Action Script /u01/app/crs/bin/racgwrap(stop) timed out for ora.rac1dr.ons! (timeout=600)
    2009-06-24 22:00:32.837: [  CRSAPP][1543690560]0StopResource error for ora.rac1dr.ons error code = -2
    2009-06-24 22:00:32.873: [  CRSRES][1554180416]0X_OP_StopResourceFailed : Stop Resource failed
    (File: rti.cpp, line: 1698
    2009-06-24 22:00:32.873: [  CRSRES][1554180416][ALERT]0`ora.rac1dr.gsd` on member `rac1dr` has experienced an unrecoverable failure.
    2009-06-24 22:00:32.873: [  CRSRES][1554180416]0Human intervention required to resume its availability.
    2009-06-24 22:00:32.921: [  CRSRES][1543690560]0X_OP_StopResourceFailed : Stop Resource failed
    (File: rti.cpp, line: 1698
    2009-06-24 22:00:32.921: [  CRSRES][1543690560][ALERT]0`ora.rac1dr.ons` on member `rac1dr` has experienced an unrecoverable failure.
    2009-06-24 22:00:32.921: [  CRSRES][1543690560]0Human intervention required to resume its availability.
    Any help is highly appreciated.
    Kamy

    CRS on RHEL 5.5 auto-start problem
    hi all
    Recently I install RAC Oracle 10gR2 on Red hat RHEL 5.5. “Two node cluster, iscsi storage, ocfs2 mounted files & ASM ”
    I did research and took all the necessary actions to install it on Red hat. Everything went well, according to the documents and at the end of the installation, all the resources were up and running. However, after the install, only the VIP was up and ONS and GSD ,ASM and the other services was in unknown state. When I stop the CRS & start it –all again, everything stars working. Please friends I need to fix this problem because the cluster services and the client workstation are stopping when the servicers is relocated to node 1 after the server reboot because the CRS did not worked still on node and manual starting is taking time.
    This is the result from crsd.log from node1 when node is online .
    Name Type Target State Host
    ora....SM1.asm      application ONLINE UNKNOWN linux1
    ora....X1.lsnr      application ONLINE UNKNOWN linux1
    ora.linux1.gsd     application ONLINE UNKNOWN linux1
    ora.linux1.ons      application ONLINE UNKNOWN linux1
    ora.linux1.vip     application ONLINE ONLINE linux1
    ora....SM2.asm     application ONLINE ONLINE linux2
    ora....X2.lsnr      application ONLINE ONLINE linux2
    ora.linux2.gsd      application ONLINE ONLINE linux2
    ora.linux2.ons     application ONLINE ONLINE linux2
    ora.linux2.vip      application ONLINE ONLINE linux2
    ora.mutdb.db      application ONLINE ONLINE linux2
    ora....b1.inst      application ONLINE OFFLINE
    ora....b2.inst      application ONLINE ONLINE linux2
    ora...._taf.cs      application ONLINE ONLINE linux2
    ora....db1.srv      application ONLINE UNKNOWN linux1
    ora....db2.srv      application ONLINE ONLINE linux2
    ############################### CRS Log ON Node 1 ##################################
    2010-05-27 11:54:57.357: [ default][3741415840][ENTER]0
    Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2004, Oracle. All rights reserved
    2010-05-27 11:54:57.366: [ default][3741415840]0CRS Daemon Starting
    2010-05-27 11:54:57.366: [ CRSMAIN][3741415840]0Checking the OCR device
    2010-05-27 11:54:57.379: [ CRSMAIN][3741415840]0Connecting to the CSS Daemon
    2010-05-27 11:54:57.783: [ COMMCRS][1107867968]clsc_connect: (0x8481ce0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_linux1_crs))
    2010-05-27 11:54:57.783: [ CSSCLNT][3741415840]clsssInitNative: connect failed, rc 9
    2010-05-27 11:54:57.783: [  CRSRTI][3741415840]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    2010-05-27 11:54:59.182: [ COMMCRS][1107867968]clsc_connect: (0x8481f60) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_linux1_crs))
    2010-05-27 11:54:59.182: [ CSSCLNT][3741415840]clsssInitNative: connect failed, rc 9
    2010-05-27 11:54:59.182: [  CRSRTI][3741415840]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    2010-05-27 11:55:00.569: [ COMMCRS][1107867968]clsc_connect: (0x84821f0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_linux1_crs))
    2010-05-27 11:55:00.569: [ CSSCLNT][3741415840]clsssInitNative: connect failed, rc 9
    2010-05-27 11:55:00.569: [  CRSRTI][3741415840]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    2010-05-27 11:55:01.954: [ COMMCRS][1107867968]clsc_connect: (0x84824b0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_linux1_crs))
    2010-05-27 11:55:01.954: [ CSSCLNT][3741415840]clsssInitNative: connect failed, rc 9
    2010-05-27 11:55:01.954: [  CRSRTI][3741415840]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
    2010-05-27 11:55:05.924: [    CRSD][3741415840]0Daemon Version: 10.2.0.1.0 Active Version: 10.2.0.1.0
    2010-05-27 11:55:05.924: [    CRSD][3741415840]0Active Version and Software Version are same
    2010-05-27 11:55:05.924: [ CRSMAIN][3741415840]0Initializing OCR
    2010-05-27 11:55:05.937: [  OCRRAW][3741415840]proprioo: for disk 0 (/u02/OCR_1.dbf), id match (1), my id set (1273042630,1936275375) total id sets (1), 1st set (1273042630,1936275375), 2nd set (0,0) my votes (1), total votes (2)
    2010-05-27 11:55:05.938: [  OCRRAW][3741415840]proprioo: for disk 1 (/u03/OCR_2.dbf), id match (1), my id set (1273042630,1936275375) total id sets (1), 1st set (1273042630,1936275375), 2nd set (0,0) my votes (1), total votes (2)
    2010-05-27 11:55:05.981: [    CRSD][3741415840]0ENV Logging level for Module: allcomp 0
    2010-05-27 11:55:05.983: [    CRSD][3741415840]0ENV Logging level for Module: default 0
    2010-05-27 11:55:05.994: [    CRSD][3741415840]0ENV Logging level for Module: COMMCRS 0
    2010-05-27 11:55:05.996: [    CRSD][3741415840]0ENV Logging level for Module: COMMNS 0
    2010-05-27 11:55:05.999: [    CRSD][3741415840]0ENV Logging level for Module: CRSUI 0
    2010-05-27 11:55:06.001: [    CRSD][3741415840]0ENV Logging level for Module: CRSCOMM 0
    2010-05-27 11:55:06.003: [    CRSD][3741415840]0ENV Logging level for Module: CRSRTI 0
    2010-05-27 11:55:06.006: [    CRSD][3741415840]0ENV Logging level for Module: CRSMAIN 0
    2010-05-27 11:55:06.008: [    CRSD][3741415840]0ENV Logging level for Module: CRSPLACE 0
    2010-05-27 11:55:06.010: [    CRSD][3741415840]0ENV Logging level for Module: CRSAPP 0
    2010-05-27 11:55:06.012: [    CRSD][3741415840]0ENV Logging level for Module: CRSRES 0
    2010-05-27 11:55:06.015: [    CRSD][3741415840]0ENV Logging level for Module: CRSOCR 0
    2010-05-27 11:55:06.017: [    CRSD][3741415840]0ENV Logging level for Module: CRSTIMER 0
    2010-05-27 11:55:06.019: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [148436880] retval lht [-27] Signal CV.
    2010-05-27 11:55:06.019: [    CRSD][3741415840]0ENV Logging level for Module: CRSEVT 0
    2010-05-27 11:55:06.022: [    CRSD][3741415840]0ENV Logging level for Module: CRSD 0
    2010-05-27 11:55:06.024: [    CRSD][3741415840]0ENV Logging level for Module: CLUCLS 0
    2010-05-27 11:55:06.027: [    CRSD][3741415840]0ENV Logging level for Module: OCRRAW 0
    2010-05-27 11:55:06.029: [    CRSD][3741415840]0ENV Logging level for Module: OCROSD 0
    2010-05-27 11:55:06.032: [    CRSD][3741415840]0ENV Logging level for Module: CSSCLNT 0
    2010-05-27 11:55:06.034: [    CRSD][3741415840]0ENV Logging level for Module: OCRAPI 0
    2010-05-27 11:55:06.036: [    CRSD][3741415840]0ENV Logging level for Module: OCRUTL 0
    2010-05-27 11:55:06.039: [    CRSD][3741415840]0ENV Logging level for Module: OCRMSG 0
    2010-05-27 11:55:06.041: [    CRSD][3741415840]0ENV Logging level for Module: OCRCLI 0
    2010-05-27 11:55:06.043: [    CRSD][3741415840]0ENV Logging level for Module: OCRCAC 0
    2010-05-27 11:55:06.046: [    CRSD][3741415840]0ENV Logging level for Module: OCRSRV 0
    2010-05-27 11:55:06.048: [    CRSD][3741415840]0ENV Logging level for Module: OCRMAS 0
    2010-05-27 11:55:06.048: [ CRSMAIN][3741415840]0Filename is /u01/app/crs/crs/init/linux1.pid
    [  clsdmt][1401583936]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=linux1DBG_CRSD))
    2010-05-27 11:55:06.119: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [149741888] retval lht [-27] Signal CV.
    2010-05-27 11:55:07.002: [ CRSMAIN][3741415840]0Using Authorizer location: /u01/app/crs/crs/auth/
    2010-05-27 11:55:07.008: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [149755728] retval lht [-27] Signal CV.
    2010-05-27 11:55:07.230: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [149740416] retval lht [-27] Signal CV.
    2010-05-27 11:55:07.380: [ CRSMAIN][3741415840]0Initializing RTI
    2010-05-27 11:55:07.380: [CRSTIMER][1422563648]0Timer Thread Starting.
    2010-05-27 11:55:07.386: [  CRSRES][3741415840]0Parameter SECURITY = 1, running in USER Mode
    2010-05-27 11:55:07.386: [ CRSMAIN][3741415840]0Initializing EVMMgr
    2010-05-27 11:55:07.453: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [150974816] retval lht [-27] Signal CV.
    2010-05-27 11:55:07.474: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [150975552] retval lht [-27] Signal CV.
    2010-05-27 11:55:07.485: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [150988032] retval lht [-27] Signal CV.
    2010-05-27 11:55:07.723: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [150998720] retval lht [-27] Signal CV.
    2010-05-27 11:55:07.734: [ CRSMAIN][3741415840]0CRSD locked during state recovery, please wait.
    2010-05-27 11:55:07.737: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [150997792] retval lht [-27] Signal CV.
    2010-05-27 11:55:07.923: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [151007072] retval lht [-27] Signal CV.
    2010-05-27 11:55:08.032: [ CRSMAIN][3741415840]0CRSD recovered, unlocked.
    2010-05-27 11:55:08.032: [ CRSMAIN][3741415840]0QS socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))
    2010-05-27 11:55:08.051: [ CRSMAIN][3741415840]0CRSD UI socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET))
    2010-05-27 11:55:08.055: [ CRSMAIN][3741415840]0E2E socket on: (ADDRESS=(PROTOCOL=tcp)(HOST=linux1-priv)(PORT=49896))
    2010-05-27 11:55:08.055: [ CRSMAIN][3741415840]0Starting Threads
    2010-05-27 11:55:08.055: [ CRSMAIN][3741415840]0CRS Daemon Started.
    2010-05-27 11:55:08.204: [  CRSRES][1527462208]0StopResource: setting CLI values
    2010-05-27 11:55:08.220: [  CRSRES][1527462208]0Attempting to stop `ora.linux1.vip` on member `linux2`
    2010-05-27 11:55:08.515: [  CRSRES][1527462208]0Stop of `ora.linux1.vip` on member `linux2` succeeded.
    2010-05-27 11:55:08.529: [  CRSRES][1527462208]0startRunnable: setting CLI values
    2010-05-27 11:55:08.529: [  CRSRES][1527462208]0Attempting to start `ora.linux1.vip` on member `linux1`
    2010-05-27 11:55:08.686: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [154566384] retval lht [-27] Signal CV.
    2010-05-27 11:55:14.000: [  CRSRES][1527462208]0Start of `ora.linux1.vip` on member `linux1` succeeded.
    2010-05-27 11:55:14.231: [  CRSRES][1527462208]0startRunnable: setting CLI values
    2010-05-27 11:55:14.272: [  CRSRES][1527462208]0Attempting to start `ora.linux1.LISTENER_LINUX1.lsnr` on member `linux1`
    2010-05-27 11:55:14.426: [  CRSEVT][1527462208]0CAAMonitorHandler :: 0:Action Script /u01/app/oracle/product/10.2.0/db_1/bin/racgwrap(start) timed out for ora.linux1.LISTENER_LINUX1.lsnr! (timeout=600)
    2010-05-27 11:55:14.426: [  CRSAPP][1527462208]0StartResource error for ora.linux1.LISTENER_LINUX1.lsnr error code = -2
    2010-05-27 11:55:14.489: [  CRSEVT][1527462208]0CAAMonitorHandler :: 0:Action Script /u01/app/oracle/product/10.2.0/db_1/bin/racgwrap(stop) timed out for ora.linux1.LISTENER_LINUX1.lsnr! (timeout=600)
    2010-05-27 11:55:14.489: [  CRSAPP][1527462208]0StopResource error for ora.linux1.LISTENER_LINUX1.lsnr error code = -2
    2010-05-27 11:55:14.495: [  CRSRES][1527462208]0X_OP_StopResourceFailed : Stop Resource failed
    (File: rti.cpp, line: 1698
    2010-05-27 11:55:14.495: [  CRSRES][1527462208][ALERT]0`ora.linux1.LISTENER_LINUX1.lsnr` on member `linux1` has experienced an unrecoverable failure.
    2010-05-27 11:55:14.495: [  CRSRES][1527462208]0Human intervention required to resume its availability.
    2010-05-27 11:55:14.628: [  CRSRES][1527462208]0startRunnable: setting CLI values
    2010-05-27 11:55:14.698: [  CRSRES][1527462208]0Attempting to start `ora.linux1.ASM1.asm` on member `linux1`
    2010-05-27 11:55:14.836: [  CRSEVT][1527462208]0CAAMonitorHandler :: 0:Action Script /u01/app/oracle/product/10.2.0/db_1/bin/racgwrap(start) timed out for ora.linux1.ASM1.asm! (timeout=600)
    2010-05-27 11:55:14.836: [  CRSAPP][1527462208]0StartResource error for ora.linux1.ASM1.asm error code = -2
    2010-05-27 11:55:14.875: [  CRSEVT][1527462208]0CAAMonitorHandler :: 0:Action Script /u01/app/oracle/product/10.2.0/db_1/bin/racgwrap(stop) timed out for ora.linux1.ASM1.asm! (timeout=600)
    2010-05-27 11:55:14.875: [  CRSAPP][1527462208]0StopResource error for ora.linux1.ASM1.asm error code = -2
    2010-05-27 11:55:14.881: [  CRSRES][1527462208]0X_OP_StopResourceFailed : Stop Resource failed
    (File: rti.cpp, line: 1698
    2010-05-27 11:55:14.881: [  CRSRES][1527462208][ALERT]0`ora.linux1.ASM1.asm` on member `linux1` has experienced an unrecoverable failure.
    2010-05-27 11:55:14.881: [  CRSRES][1527462208]0Human intervention required to resume its availability.
    2010-05-27 11:55:15.051: [  CRSRES][1527462208]0CRS-1028: Dependency analysis failed because of:
    'Resource in UNKNOWN state: ora.linux1.ASM1.asm'
    2010-05-27 11:55:15.185: [  CRSRES][1527462208]0startRunnable: setting CLI values
    2010-05-27 11:55:15.237: [  CRSRES][1537952064]0startRunnable: setting CLI values
    2010-05-27 11:55:15.306: [  CRSRES][1558931776]0startRunnable: setting CLI values
    2010-05-27 11:55:15.339: [  CRSRES][1527462208]0Attempting to start `ora.linux1.gsd` on member `linux1`
    2010-05-27 11:55:15.397: [  CRSRES][1537952064]0Attempting to start `ora.linux1.ons` on member `linux1`
    2010-05-27 11:55:15.407: [  CRSRES][1558931776]0Attempting to start `ora.mutdb.mutdb_taf.mutdb1.srv` on member `linux1`
    2010-05-27 11:55:15.789: [  CRSEVT][1527462208]0CAAMonitorHandler :: 0:Action Script /u01/app/crs/bin/racgwrap(start) timed out for ora.linux1.gsd! (timeout=600)
    2010-05-27 11:55:15.789: [  CRSAPP][1527462208]0StartResource error for ora.linux1.gsd error code = -2
    2010-05-27 11:55:15.808: [  CRSEVT][1537952064]0CAAMonitorHandler :: 0:Action Script /u01/app/crs/bin/racgwrap(start) timed out for ora.linux1.ons! (timeout=600)
    2010-05-27 11:55:15.808: [  CRSAPP][1537952064]0StartResource error for ora.linux1.ons error code = -2
    2010-05-27 11:55:15.812: [  CRSEVT][1558931776]0CAAMonitorHandler :: 0:Action Script /u01/app/oracle/product/10.2.0/db_1/bin/racgwrap(start) timed out for ora.mutdb.mutdb_taf.mutdb1.srv! (timeout=600)
    2010-05-27 11:55:15.812: [  CRSAPP][1558931776]0StartResource error for ora.mutdb.mutdb_taf.mutdb1.srv error code = -2
    2010-05-27 11:55:15.838: [  CRSEVT][1527462208]0CAAMonitorHandler :: 0:Action Script /u01/app/crs/bin/racgwrap(stop) timed out for ora.linux1.gsd! (timeout=600)
    2010-05-27 11:55:15.838: [  CRSAPP][1527462208]0StopResource error for ora.linux1.gsd error code = -2
    2010-05-27 11:55:15.844: [  CRSEVT][1537952064]0CAAMonitorHandler :: 0:Action Script /u01/app/crs/bin/racgwrap(stop) timed out for ora.linux1.ons! (timeout=600)
    2010-05-27 11:55:15.845: [  CRSAPP][1537952064]0StopResource error for ora.linux1.ons error code = -2
    2010-05-27 11:55:15.849: [  CRSRES][1527462208]0X_OP_StopResourceFailed : Stop Resource failed
    (File: rti.cpp, line: 1698
    2010-05-27 11:55:15.849: [  CRSRES][1527462208][ALERT]0`ora.linux1.gsd` on member `linux1` has experienced an unrecoverable failure.
    2010-05-27 11:55:15.849: [  CRSRES][1527462208]0Human intervention required to resume its availability.
    2010-05-27 11:55:15.859: [  CRSEVT][1558931776]0CAAMonitorHandler :: 0:Action Script /u01/app/oracle/product/10.2.0/db_1/bin/racgwrap(stop) timed out for ora.mutdb.mutdb_taf.mutdb1.srv! (timeout=600)
    2010-05-27 11:55:15.860: [  CRSAPP][1558931776]0StopResource error for ora.mutdb.mutdb_taf.mutdb1.srv error code = -2
    2010-05-27 11:55:15.863: [  CRSRES][1537952064]0X_OP_StopResourceFailed : Stop Resource failed
    (File: rti.cpp, line: 1698
    2010-05-27 11:55:15.863: [  CRSRES][1537952064][ALERT]0`ora.linux1.ons` on member `linux1` has experienced an unrecoverable failure.
    2010-05-27 11:55:15.863: [  CRSRES][1537952064]0Human intervention required to resume its availability.
    2010-05-27 11:55:15.869: [  CRSRES][1558931776]0X_OP_StopResourceFailed : Stop Resource failed
    (File: rti.cpp, line: 1698
    2010-05-27 11:55:15.869: [  CRSRES][1558931776][ALERT]0`ora.mutdb.mutdb_taf.mutdb1.srv` on member `linux1` has experienced an unrecoverable failure.
    2010-05-27 11:55:15.869: [  CRSRES][1558931776]0Human intervention required to resume its availability.
    2010-05-27 11:55:20.450: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [154578560] retval lht [-27] Signal CV.
    2010-05-27 11:55:21.130: [  OCRSRV][1223256384]th_select_handler: Failed to retrieve procctx from ht. constr = [154578560] retval lht [-27] Signal CV.
    I tired this solution it was on website “but it did not work”
    下面是METALINK给出的解释：
    I have checked the information provided. crsd is reporting timeouts when running the checking actions. The racgmain process on the other hand, executed by racgwrap, remain running so the number is increasing.This issue has been reported in Bug 7009245 ""RACGMAIN CHECK" PROCESS NOT TERMINATING", and drill down and determine the root cause this require to get a stack trace from racgmain process(es) as well as you should runOSWatcher with a specific parameters, finally the issue of Bug 7009245 has been fixed by the fix of Bug 6196746 (unpublished bug -- summary is "racgwrap spawns racgmain child process. crsd timeout kills racgwrap but bot racgmain")So please perform the following action plan that intend to solve the issue.
    .ACTION PLAN
    ============
    1. Stop CRS on this node.
    2. Make a copy of racgwrap located under $ORACLE_HOME/bin and $CRS_HOME/bin
    3. Edit the file racgwrap and modify the last 3 lines from:
    .$ORACLE_HOME/bin/racgmain "$@"
    status=$?
    exit $status
    .to:
    .# Line added to test fix for Bug 6196746
    exec $ORACLE_HOME/bin/racgmain "$@"
    .4. Restart CRS and make sure that all the resources are startes
    This should solve the issue. also if you are not able to stop CRS right now, you can try the
    solution without crs stop/start.
    BUT IT DID NOT WORK
    ###################################################3

  • Crs stop and acfs filesystem

    i have created acfs file system and mounted for 2 node RAC.
    when i issued crsctl stop crs for local node, the asm instance could not stop.
    asm alert log showed that diskgroup for acfs and voting disk could not be dismounted as they are used by processes
    realising this i did unmount the acfs filesystem, then dismounted acfs diskgroup manually but the diskgroup with voting disk on it now showed status quiesced,
    eventually i shut abort the asm instance.
    is it required to umount the acfs explicitly prior to stopping crs, doesnt clusterware offline/stop acfs filesystem as it does for the other crs resources.

    no,vote/ocr is not stored on acfs.
    ORA-15027 active use of diskgroup "ACFS" precludes its dismount
    ORA-15027 active use of diskgroup "VOTE" precludes its dismount
    question: is the foll understanding correct
    it is required to unmount the acfs filesystem explicitly prior to stopping crs.
    because crsctl stop crs wont offline/stop acfs filesystem as it does for the other crs resources.

  • Root.ah fails on 2nd node(rac2) with [ ORA-15018,ORA-15017,ORA-15003 ]

    Hi All,
    I m trying to setup 11gR2 Grid installation on two-node Rac . When it comes to running root.sh on second node (i.e. rac2) it fails with below error. Could please anyone help me out. This is my 3rd attempt and all fails with below errors on node 2.
    rac2:
    [root@rac2 grid_home]# ./root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= grid
        ORACLE_HOME=  /u01/grid_home
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
       Copying dbhome to /usr/local/bin ...
       Copying oraenv to /usr/local/bin ...
       Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2013-07-10 18:53:15: Parsing the host name
    2013-07-10 18:53:15: Checking for super user privileges
    2013-07-10 18:53:15: User has super user privileges
    Using configuration parameter file: /u01/grid_home/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
    CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
    DiskGroup CRS creation failed with the following message:
    ORA-15018: diskgroup cannot be created
    ORA-15017: diskgroup "CRS" cannot be mounted
    ORA-15003: diskgroup "CRS" already mounted in another lock name space
    Configuration of ASM failed, see logs for details
    Did not succssfully configure and start ASM
    CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/grid_home/bin/crsctl stop resource ora.crsd -init
    Stop of resource "ora.crsd -init" failed
    Failed to stop CRSD
    CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
    CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
    CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac2'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
    CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
    CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
    CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
    CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
    Initial cluster configuration failed.  See /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log for details
    [root@rac2 grid_home]#
    rac2  alertrac2.log
    [root@rac2 rac2]# cat -n alertrac2.log
         1  Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
         2  2013-07-10 18:53:16.145
         3  [client(13088)]CRS-2106:The OLR location /u01/grid_home/cdata/rac2.olr is inaccessible. Details in /u01/grid_home/log/rac2/client/ocrconfig_13088.log.
         4  2013-07-10 18:53:16.228
         5  [client(13088)]CRS-2101:The OLR was formatted using version 3.
         6  2013-07-10 18:53:31.734
         7  [ohasd(13132)]CRS-2112:The OLR service started on node rac2.
         8  2013-07-10 18:53:31.893
         9  [ohasd(13132)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
        10  2013-07-10 18:53:53.762
        11  [ohasd(13132)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
        12  2013-07-10 18:53:55.381
        13  [cssd(14409)]CRS-1713:CSSD daemon is started in exclusive mode
        14  2013-07-10 18:54:01.530
        15  [cssd(14409)]CRS-1709:Lease acquisition failed for node rac2 because no voting file has been configured; Details at (:CSSNM00031:) in /u01/grid_home/log/rac2/cssd/ocssd.log
        16  2013-07-10 18:54:19.113
        17  [cssd(14409)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
        18  2013-07-10 18:54:19.910
        19  [ctssd(14465)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
        20  2013-07-10 18:54:19.920
        21  [ctssd(14465)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
        22  2013-07-10 18:54:20.903
        23  [ctssd(14465)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
        24  [client(14715)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
        25  [client(14719)]CRS-10001:ACFS-9322: done.
        26  2013-07-10 18:54:47.104
        27  [ctssd(14465)]CRS-2405:The Cluster Time Synchronization Service on host rac2 is shutdown by user
        28  2013-07-10 18:54:55.837
        29  [cssd(14409)]CRS-1603:CSSD on node rac2 shutdown by user.
    rac2 rootcrs logfile
    [root@rac2 rac2]# cat  /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log
    2013-07-10 18:53:15: The configuration parameter file /u01/grid_home/crs/install/crsconfig_params is valid
    2013-07-10 18:53:15: Checking for super user privileges
    2013-07-10 18:53:15: User has super user privileges
    2013-07-10 18:53:15: ### Printing the configuration values from files:
    2013-07-10 18:53:15:    /u01/grid_home/crs/install/crsconfig_params
    2013-07-10 18:53:15:    /u01/grid_home/crs/install/s_crsconfig_defs
    2013-07-10 18:53:15: ASM_DISCOVERY_STRING=
    2013-07-10 18:53:15: ASM_DISKS=ORCL:CRS1
    2013-07-10 18:53:15: ASM_DISK_GROUP=CRS
    2013-07-10 18:53:15: ASM_REDUNDANCY=EXTERNAL
    2013-07-10 18:53:15: ASM_SPFILE=
    2013-07-10 18:53:15: ASM_UPGRADE=false
    2013-07-10 18:53:15: CLSCFG_MISSCOUNT=
    2013-07-10 18:53:15: CLUSTER_GUID=
    2013-07-10 18:53:15: CLUSTER_NAME=rac-scan
    2013-07-10 18:53:15: CRS_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'
    2013-07-10 18:53:15: CRS_STORAGE_OPTION=1
    2013-07-10 18:53:15: CSS_LEASEDURATION=400
    2013-07-10 18:53:15: DIRPREFIX=
    2013-07-10 18:53:15: DISABLE_OPROCD=0
    2013-07-10 18:53:15: EMBASEJAR_NAME=oemlt.jar
    2013-07-10 18:53:15: EWTJAR_NAME=ewt3.jar
    2013-07-10 18:53:15: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
    2013-07-10 18:53:15: GNS_ADDR_LIST=
    2013-07-10 18:53:15: GNS_ALLOW_NET_LIST=
    2013-07-10 18:53:15: GNS_CONF=false
    2013-07-10 18:53:15: GNS_DENY_ITF_LIST=
    2013-07-10 18:53:15: GNS_DENY_NET_LIST=
    2013-07-10 18:53:15: GNS_DOMAIN_LIST=
    2013-07-10 18:53:15: GPNPCONFIGDIR=/u01/grid_home
    2013-07-10 18:53:15: GPNPGCONFIGDIR=/u01/grid_home
    2013-07-10 18:53:15: GPNP_PA=
    2013-07-10 18:53:15: HELPJAR_NAME=help4.jar
    2013-07-10 18:53:15: HOST_NAME_LIST=rac1,rac2
    2013-07-10 18:53:15: ID=/etc/init.d
    2013-07-10 18:53:15: INIT=/sbin/init
    2013-07-10 18:53:15: IT=/etc/inittab
    2013-07-10 18:53:15: JEWTJAR_NAME=jewt4.jar
    2013-07-10 18:53:15: JLIBDIR=/u01/grid_home/jlib
    2013-07-10 18:53:15: JREDIR=/u01/grid_home/jdk/jre/
    2013-07-10 18:53:15: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
    2013-07-10 18:53:15: MSGFILE=/var/adm/messages
    2013-07-10 18:53:15: NETCFGJAR_NAME=netcfg.jar
    2013-07-10 18:53:15: NETWORKS="eth0"/192.168.0.0:public,"eth1"/192.168.1.0:cluster_interconnect
    2013-07-10 18:53:15: NEW_HOST_NAME_LIST=
    2013-07-10 18:53:15: NEW_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'
    2013-07-10 18:53:15: NEW_NODE_NAME_LIST=
    2013-07-10 18:53:15: NEW_PRIVATE_NAME_LIST=
    2013-07-10 18:53:15: NODELIST=rac1,rac2
    2013-07-10 18:53:15: NODE_NAME_LIST=rac1,rac2
    2013-07-10 18:53:15: OCFS_CONFIG=
    2013-07-10 18:53:15: OCRCONFIG=/etc/oracle/ocr.loc
    2013-07-10 18:53:15: OCRCONFIGDIR=/etc/oracle
    2013-07-10 18:53:15: OCRID=
    2013-07-10 18:53:15: OCRLOC=ocr.loc
    2013-07-10 18:53:15: OCR_LOCATIONS=NO_VAL
    2013-07-10 18:53:15: OLASTGASPDIR=/etc/oracle/lastgasp
    2013-07-10 18:53:15: OLRCONFIG=/etc/oracle/olr.loc
    2013-07-10 18:53:15: OLRCONFIGDIR=/etc/oracle
    2013-07-10 18:53:15: OLRLOC=olr.loc
    2013-07-10 18:53:15: OPROCDCHECKDIR=/etc/oracle/oprocd/check
    2013-07-10 18:53:15: OPROCDDIR=/etc/oracle/oprocd
    2013-07-10 18:53:15: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
    2013-07-10 18:53:15: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
    2013-07-10 18:53:15: ORACLE_BASE=/u01/11.2.0
    2013-07-10 18:53:15: ORACLE_HOME=/u01/grid_home
    2013-07-10 18:53:15: ORACLE_OWNER=grid
    2013-07-10 18:53:15: ORA_ASM_GROUP=asmadmin
    2013-07-10 18:53:15: ORA_DBA_GROUP=oinstall
    2013-07-10 18:53:15: PRIVATE_NAME_LIST=
    2013-07-10 18:53:15: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
    2013-07-10 18:53:15: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
    2013-07-10 18:53:15: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
    2013-07-10 18:53:15: RC_KILL=K19
    2013-07-10 18:53:15: RC_KILL_OLD=K96
    2013-07-10 18:53:15: RC_START=S96
    2013-07-10 18:53:15: SCAN_NAME=rac-scan.naveed.com
    2013-07-10 18:53:15: SCAN_PORT=1521
    2013-07-10 18:53:15: SCRBASE=/etc/oracle/scls_scr
    2013-07-10 18:53:15: SHAREJAR_NAME=share.jar
    2013-07-10 18:53:15: SILENT=false
    2013-07-10 18:53:15: SO_EXT=so
    2013-07-10 18:53:15: SRVCFGLOC=srvConfig.loc
    2013-07-10 18:53:15: SRVCONFIG=/var/opt/oracle/srvConfig.loc
    2013-07-10 18:53:15: SRVCONFIGDIR=/var/opt/oracle
    2013-07-10 18:53:15: VNDR_CLUSTER=false
    2013-07-10 18:53:15: VOTING_DISKS=NO_VAL
    2013-07-10 18:53:15: ### Printing other configuration values ###
    2013-07-10 18:53:15: CLSCFG_EXTRA_PARMS=
    2013-07-10 18:53:15: CRSDelete=0
    2013-07-10 18:53:15: CRSPatch=0
    2013-07-10 18:53:15: DEBUG=
    2013-07-10 18:53:15: DOWNGRADE=
    2013-07-10 18:53:15: HAS_GROUP=oinstall
    2013-07-10 18:53:15: HAS_USER=root
    2013-07-10 18:53:15: HOST=rac2
    2013-07-10 18:53:15: IS_SIHA=0
    2013-07-10 18:53:15: OLR_DIRECTORY=/u01/grid_home/cdata
    2013-07-10 18:53:15: OLR_LOCATION=/u01/grid_home/cdata/rac2.olr
    2013-07-10 18:53:15: ORA_CRS_HOME=/u01/grid_home
    2013-07-10 18:53:15: SUPERUSER=root
    2013-07-10 18:53:15: UPGRADE=
    2013-07-10 18:53:15: VF_DISCOVERY_STRING=
    2013-07-10 18:53:15: addfile=/u01/grid_home/crs/install/crsconfig_addparams
    2013-07-10 18:53:15: crscfg_trace=1
    2013-07-10 18:53:15: crscfg_trace_file=/u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log
    2013-07-10 18:53:15: hosts=
    2013-07-10 18:53:15: oldcrshome=
    2013-07-10 18:53:15: oldcrsver=
    2013-07-10 18:53:15: osdfile=/u01/grid_home/crs/install/s_crsconfig_defs
    2013-07-10 18:53:15: parameters_valid=1
    2013-07-10 18:53:15: paramfile=/u01/grid_home/crs/install/crsconfig_params
    2013-07-10 18:53:15: platform_family=unix
    2013-07-10 18:53:15: srvctl_trc_suff=0
    2013-07-10 18:53:15: unlock_crshome=
    2013-07-10 18:53:15: user_is_superuser=1
    2013-07-10 18:53:15: ### Printing of configuration values complete ###
    2013-07-10 18:53:15: Oracle CRS stack is not configured yet
    2013-07-10 18:53:15: CRS is not yet configured. Hence, will proceed to configure CRS
    2013-07-10 18:53:15: Cluster-wide one-time actions... Done!
    2013-07-10 18:53:15: Oracle CRS home = /u01/grid_home
    2013-07-10 18:53:15: Host name = rac2
    2013-07-10 18:53:15: CRS user = grid
    2013-07-10 18:53:15: Oracle CRS home = /u01/grid_home
    2013-07-10 18:53:15: GPnP host = rac2
    2013-07-10 18:53:15: Oracle GPnP home = /u01/grid_home/gpnp
    2013-07-10 18:53:15: Oracle GPnP local home = /u01/grid_home/gpnp/rac2
    2013-07-10 18:53:15: GPnP directories verified.
    2013-07-10 18:53:15: Checking to see if Oracle CRS stack is already configured
    2013-07-10 18:53:15: Oracle CRS stack is not configured yet
    2013-07-10 18:53:15: ---Checking local gpnp setup...
    2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml" does not exist
    2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso" does not exist
    2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso" does not exist
    2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp/rac2: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
    2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp/rac2: INVALID (bad profile/wallet)
    2013-07-10 18:53:15: ---Checking cluster-wide gpnp setup...
    2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1
    2013-07-10 18:53:15: gpnptool: run /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer
    2013-07-10 18:53:15: Running as user grid: /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer
    2013-07-10 18:53:15: s_run_as_user2: Running /bin/su grid -c ' /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer '
    2013-07-10 18:53:15: Removing file /tmp/file0qKE0c
    2013-07-10 18:53:15: Successfully removed file: /tmp/file0qKE0c
    2013-07-10 18:53:15: /bin/su successfully executed
    2013-07-10 18:53:15: gpnptool: rc=0
    2013-07-10 18:53:15: gpnptool output:
    Profile signature is valid.
    2013-07-10 18:53:15: Profile "/u01/grid_home/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/grid_home/gpnp/wallets/peer"
    2013-07-10 18:53:15: gpnptool: run /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer
    2013-07-10 18:53:15: Running as user grid: /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer
    2013-07-10 18:53:15: s_run_as_user2: Running /bin/su grid -c ' /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer '
    2013-07-10 18:53:16: Removing file /tmp/filebkOtBv
    2013-07-10 18:53:16: Successfully removed file: /tmp/filebkOtBv
    2013-07-10 18:53:16: /bin/su successfully executed
    2013-07-10 18:53:16: gpnptool: rc=0
    2013-07-10 18:53:16: gpnptool output:
    Profile signature is valid.
    2013-07-10 18:53:16: Profile "/u01/grid_home/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/grid_home/gpnp/wallets/prdr"
    2013-07-10 18:53:16: chk gpnphome /u01/grid_home/gpnp: OK
    2013-07-10 18:53:16: GPnP Wallets ownership/permissions successfully set.
    2013-07-10 18:53:16: gpnp setup checked: local valid? 0 cluster-wide valid? 1
    2013-07-10 18:53:16: Taking cluster-wide setup as local
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/profiles/peer/profile.xml" => "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/peer/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/prdr/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/profiles/peer/profile_orig.xml" => "/u01/grid_home/gpnp/rac2/profiles/peer/profile_orig.xml"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/profiles/peer/profile_orig.xml" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/root/ewallet.p12" => "/u01/grid_home/gpnp/rac2/wallets/root/ewallet.p12"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/root/ewallet.p12" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/pa/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/pa/cwallet.sso"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/pa/cwallet.sso" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/root/b64certificate.txt" => "/u01/grid_home/gpnp/rac2/wallets/root/b64certificate.txt"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/root/b64certificate.txt" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/peer/cert.txt" => "/u01/grid_home/gpnp/rac2/wallets/peer/cert.txt"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/peer/cert.txt" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/pa/cert.txt" => "/u01/grid_home/gpnp/rac2/wallets/pa/cert.txt"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/pa/cert.txt" => (grid,oinstall)
    2013-07-10 18:53:16: GPnP Wallets ownership/permissions successfully set.
    2013-07-10 18:53:16: gpnp setup: GOTCLUSTERWIDE
    2013-07-10 18:53:16: Validating for SI-CSS configuration
    2013-07-10 18:53:16: Retrieving OCR main disk location
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc
    2013-07-10 18:53:16: Unable to retrieve ocr disk info
    2013-07-10 18:53:16: Checking to see if any 9i GSD is up
    2013-07-10 18:53:16: libskgxnBase_lib = /etc/ORCLcluster/oracm/lib/libskgxn2.so
    2013-07-10 18:53:16: libskgxn_lib = /opt/ORCLcluster/lib/libskgxn2.so
    2013-07-10 18:53:16: SKGXN library file does not exists
    2013-07-10 18:53:16: OLR location = /u01/grid_home/cdata/rac2.olr
    2013-07-10 18:53:16: Oracle CRS Home = /u01/grid_home
    2013-07-10 18:53:16: Validating /etc/oracle/olr.loc file for OLR location /u01/grid_home/cdata/rac2.olr
    2013-07-10 18:53:16: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig
    2013-07-10 18:53:16: Oracle CRS home = /u01/grid_home
    2013-07-10 18:53:16: Oracle cluster name = rac-scan
    2013-07-10 18:53:16: OCR locations = +CRS
    2013-07-10 18:53:16: Validating OCR
    2013-07-10 18:53:16: Retrieving OCR location used by previous installations
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrmirrorconfig_loc
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc3
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc4
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc5
    2013-07-10 18:53:16: Checking if OCR sync file exists
    2013-07-10 18:53:16: No need to sync OCR file
    2013-07-10 18:53:16: OCR_LOCATION=+CRS
    2013-07-10 18:53:16: OCR_MIRROR_LOCATION=
    2013-07-10 18:53:16: OCR_MIRROR_LOC3=
    2013-07-10 18:53:16: OCR_MIRROR_LOC4=
    2013-07-10 18:53:16: OCR_MIRROR_LOC5=
    2013-07-10 18:53:16: Current OCR location=
    2013-07-10 18:53:16: Current OCR mirror location=
    2013-07-10 18:53:16: Current OCR mirror loc3=
    2013-07-10 18:53:16: Current OCR mirror loc4=
    2013-07-10 18:53:16: Current OCR mirror loc5=
    2013-07-10 18:53:16: Verifying current OCR settings with user entered values
    2013-07-10 18:53:16: Setting OCR locations in /etc/oracle/ocr.loc
    2013-07-10 18:53:16: Validating OCR locations in /etc/oracle/ocr.loc
    2013-07-10 18:53:16: Checking for existence of /etc/oracle/ocr.loc
    2013-07-10 18:53:16: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig
    2013-07-10 18:53:16: Setting ocr location +CRS
    2013-07-10 18:53:16: Creating or upgrading Oracle Local Registry (OLR)
    2013-07-10 18:53:16: OLR successfully created or upgraded
    2013-07-10 18:53:16: /u01/grid_home/bin/clscfg -localadd
    2013-07-10 18:53:16: Keys created in the OLR successfully
    2013-07-10 18:53:16: GPnP setup state: new-cluster-wide
    2013-07-10 18:53:16: GPnP cluster configuration already performed
    2013-07-10 18:53:16: Registering ohasd
    2013-07-10 18:53:16: init file = /u01/grid_home/crs/init/init.ohasd
    2013-07-10 18:53:16: Copying file /u01/grid_home/crs/init/init.ohasd to /etc/init.d directory
    2013-07-10 18:53:16: Setting init.ohasd permission in /etc/init.d directory
    2013-07-10 18:53:16: init file = /u01/grid_home/crs/init/ohasd
    2013-07-10 18:53:16: Copying file /u01/grid_home/crs/init/ohasd to /etc/init.d directory
    2013-07-10 18:53:16: Setting ohasd permission in /etc/init.d directory
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc3.d/S96ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc3.d/S96ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc5.d/S96ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc5.d/S96ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc0.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc0.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc1.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc1.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc2.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc2.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc4.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc4.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc6.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc6.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: The file ohasd has been successfully linked to the RC directories
    2013-07-10 18:53:16: Starting ohasd
    2013-07-10 18:53:16: itab entries=
    2013-07-10 18:53:21: Created backup /etc/inittab.no_crs
    2013-07-10 18:53:21: Appending to /etc/inittab.tmp:
    2013-07-10 18:53:21: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
    2013-07-10 18:53:21: Done updating /etc/inittab.tmp
    2013-07-10 18:53:21: Saved /etc/inittab.crs
    2013-07-10 18:53:21: Installed new /etc/inittab
    2013-07-10 18:53:36: ohasd is starting
    2013-07-10 18:53:36: Checking ohasd
    2013-07-10 18:53:37: ohasd started successfully
    2013-07-10 18:53:37: Creating CRS resources and dependencies
    2013-07-10 18:53:37: Configuring HASD
    2013-07-10 18:53:37: Registering type ora.daemon.type
    2013-07-10 18:53:37: Registering type ora.mdns.type
    2013-07-10 18:53:37: Registering type ora.gpnp.type
    2013-07-10 18:53:38: Registering type ora.gipc.type
    2013-07-10 18:53:38: Registering type ora.cssd.type
    2013-07-10 18:53:38: Registering type ora.cssdmonitor.type
    2013-07-10 18:53:39: Registering type ora.crs.type
    2013-07-10 18:53:39: Registering type ora.evm.type
    2013-07-10 18:53:39: Registering type ora.ctss.type
    2013-07-10 18:53:40: Registering type ora.asm.type
    2013-07-10 18:53:40: Registering type ora.drivers.acfs.type
    2013-07-10 18:53:40: Registering type ora.diskmon.type
    2013-07-10 18:53:51: ADVM/ACFS is configured
    2013-07-10 18:53:51: Successfully created CRS resources for cluster daemon and ASM
    2013-07-10 18:53:51: Checking if initial configuration has been performed
    2013-07-10 18:53:51: Starting CSS in exclusive mode
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    2013-07-10 18:54:19: Querying for existing CSS voting disks
    2013-07-10 18:54:19: Performing initial configuration for cluster
    2013-07-10 18:54:21: Start of resource "ora.ctssd -init" Succeeded
    2013-07-10 18:54:21: Configuring ASM via ASMCA
    2013-07-10 18:54:21: Executing as grid: /u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM
    2013-07-10 18:54:21: Running as user grid: /u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM
    2013-07-10 18:54:21:   Invoking "/u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM" as user "grid"
    2013-07-10 18:54:40: Configuration of ASM failed, see logs for details
    2013-07-10 18:54:40: Did not succssfully configure and start ASM
    2013-07-10 18:54:40: Exiting exclusive mode
    2013-07-10 18:54:40: Command return code of 1 (256) from command: /u01/grid_home/bin/crsctl stop resource ora.crsd -init
    2013-07-10 18:54:40: Stop of resource "ora.crsd -init" failed
    2013-07-10 18:54:40: Failed to stop CRSD
    2013-07-10 18:55:04: Initial cluster configuration failed.  See /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log for details
    Also below are some of the configs related to rac2 node
    [root@rac2 rac2]# rpm -qa | grep oracleasm
    oracleasmlib-2.0.4-1.el5
    oracleasm-support-2.1.8-1.el5
    oracleasm-2.6.18-274.el5xen-2.0.5-1.el5
    oracleasm-2.6.18-274.el5-2.0.5-1.el5
    oracleasm-2.6.18-274.el5debug-2.0.5-1.el5
    oracleasm-2.6.18-274.el5-debuginfo-2.0.5-1.el5
    [root@rac2 rac2]# /usr/sbin/oracleasm configure
    ORACLEASM_ENABLED=true
    ORACLEASM_UID=grid
    ORACLEASM_GID=asmadmin
    ORACLEASM_SCANBOOT=true
    ORACLEASM_SCANORDER=""
    ORACLEASM_SCANEXCLUDE=""
    ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"
    [root@rac2 rac2]# /usr/sbin/oracleasm status
    Checking if ASM is loaded: yes
    Checking if /dev/oracleasm is mounted: yes
    [root@rac2 rac2]# /usr/sbin/oracleasm listdisks
    CRS1
    DATA1
    FRA1
    [root@rac2 rac2]# ls -l /dev/oracleasm/disks/
    total 0
    brw-rw---- 1 grid asmadmin 8, 17 Jul 10 18:35 CRS1
    brw-rw---- 1 grid asmadmin 8, 33 Jul 10 18:36 DATA1
    brw-rw---- 1 grid asmadmin 8, 49 Jul 10 18:36 FRA1
    [root@rac2 rac2]# cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1               localhost.localdomain localhost
    ::1             localhost6.localdomain6 localhost6
    #Public IP's(eth0)
    192.168.0.101    rac1.naveed.com    rac1
    192.168.0.102    rac2.naveed.com    rac2
    #Private IP's(eth1)
    192.168.1.101    rac1-prv.naveed.com   rac1-prv
    192.168.1.102    rac2-prv.naveed.com   rac2-prv
    #VIPS
    192.168.0.221    rac1-vip.naveed.com   rac1-vip
    192.168.0.222    rac2-vip.naveed.com   rac2-vip
    #DNS server IP
    192.168.0.10    naveeddns.naveed.com   naveeddns
    [root@rac2 rac2]#
    Thanks in advance

    Hi,
    First of all thanks a lot for the response. You wont't beleive this is my 7th fresh installation and everytime in node 2 i m hit with this same error.
    Also i tried below procedure instead of fresh installation
    once i deconfig & rerun (./rootcrs.pl -verbose -deconfig -force) on node 2
    Using configuration parameter file: ./crsconfig_params
    PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
    PRCR-1068 : Failed to query resources
    Cannot communicate with crsd
    PRCR-1070 : Failed to check if resource ora.gsd is registered
    Cannot communicate with crsd
    PRCR-1070 : Failed to check if resource ora.ons is registered
    Cannot communicate with crsd
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4000: Command Stop failed, or completed with errors.
    CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
    CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac2'
    CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
    CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
    CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'
    CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
    CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
    CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
    CRS-2677: Stop of 'ora.drivers.acfs' on 'rac2' succeeded
    CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
    CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
    CRS-4133: Oracle High Availability Services has been stopped.
    Successfully deconfigured Oracle clusterware stack on this node
    [root@rac2 grid_home]# ./root.sh
    Performing root user operation for Oracle 11g
    The following environment variables are set as:
        ORACLE_OWNER= grid
        ORACLE_HOME=  /u01/grid_home
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
    The contents of "dbhome" have not changed. No need to overwrite.
    The contents of "oraenv" have not changed. No need to overwrite.
    The contents of "coraenv" have not changed. No need to overwrite.
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root script.
    Now product-specific root actions will be performed.
    Using configuration parameter file: /u01/grid_home/crs/install/crsconfig_params
    User ignored Prerequisites during installation
    OLR initialization - successful
    Adding Clusterware entries to inittab
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    ASM created and started successfully.
    Disk Group CRS mounted successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Successful addition of voting disk 636af26485ef4f27bfec31523aaa0660.
    Successfully replaced voting disk group with +CRS.
    CRS-4266: Voting file(s) successfully replaced
    ##  STATE    File Universal Id                File Name Disk group
    1. ONLINE   636af26485ef4f27bfec31523aaa0660 (ORCL:CRS1) [CRS]
    Located 1 voting disk(s).
    Start of resource "ora.crsd" failed
    CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'rac2'
    CRS-4000: Command Start failed, or completed with errors.
    Failed to start Oracle Grid Infrastructure stack
    Failed to start Cluster Ready Services at /u01/grid_home/crs/install/crsconfig_lib.pm line 1286.
    /u01/grid_home/perl/bin/perl -I/u01/grid_home/perl/lib -I/u01/grid_home/crs/install /u01/grid_home/crs/install/rootcrs.pl execution failed

Maybe you are looking for

  • Update statement getting ORA-00904 invalid identifier

    I am getting this ORA error when I try to conduct an update statement from a column value in one table to a like-named column in another table. For instance, in my select, shown below, this query runs like a champ, returning all of my records SELECT

  • Why do I need to install RSAT on Windows 8.1 for accessing Hyper-V Server 2012 R2?

    Since Windows 8.1 has Hyper-V Platform and Hyper-V Manager built in to it, if I just enable the Hyper-V Manager on my Windows 8.1 client machine I should be able to access the standalone Hyper-V Server 2012 R2 (which I'm planning to install on my 2nd

  • How to handle multiple connection or user with l2cap

    hi friends, I need your help please a.s.a.p how to handle multiple connection with l2cap protocol in j2me sorry, I'm a new programmer in j2me thanks all...

  • Garnishment Refund - Credit Memo

    Hello experts, I have the following scenario  of Garnishment refund - An employee was garnished $54.75 which generated check number . Vendor returned the check as the Garishment was satisfied.A/P has voided this check and a refund has been created in

  • Customer exit in BEx to derive Cal month

    hey all, i need to write  a customer exit, to derive  prev Cal Month value from the given Calday. I have written the code but it derives the current cal month instead of previous. any suggestions ? case I_VNAM.   when 'ZCU_CMTH' .     if i_step = 2.