Ora.asm -init failed on second node root.sh

Hi All,
Installing Grid Infrastructure for a 11gr2 Cluster on two nodes Oracle Linux 5 + Vsware vSphere v4, shared disk on same host machine. When run root.sh, first node was success but the second node got following error message (actually the first node was cloned from the seoncd):
CRS-2672: Attempting to start 'ora.ctssd' on 'wandrac2'
Start action for octssd aborted
CRS-2676: Start of 'ora.ctssd' on 'wandrac2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'wandrac2'
CRS-2672: Attempting to start 'ora.asm' on 'wandrac2'
CRS-2676: Start of 'ora.drivers.acfs' on 'wandrac2' succeeded
CRS-2676: Start of 'ora.asm' on 'wandrac2' succeeded
CRS-2664: Resource 'ora.ctssd' is already running on 'wandrac2'
CRS-4000: Command Start failed, or completed with errors.
Command return code of 1 (256) from command: /orapp/racsl/11.2.0/bin/crsctl start resource ora.asm -init
Start of resource "ora.asm -init" failed
Failed to start ASM
Failed to start Oracle Clusterware stack
Thanks in advance for any information and helps,

Hi,
I came across this error and I am about to start a fresh installation of the grid. (ealier one failed because it was unable to read the memory in rac2 )
Is there anything specific I can change before I start my installation.
PS - I didnt get what exactly is going on with the hosts file.
My files are as follows :
RAC1 - etc/hosts
[oracle@falcen6a ~]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
# Public
192.168.100.218 falcen6a.a.pri falcen6a
192.168.100.219 falcen6b.a.pri falcen6b
# Private
192.168.210.101 falcen6a-priv.a.pri falcen6a-priv
192.168.210.102 falcen6b-priv.a.pri falcen6b-priv
# Virtual
192.168.100.212 falcen6a-vip.a.pri falcen6a-vip
192.168.100.213 falcen6b-vip.a.pri falcen6b-vip
# SCAN
#192.168.100.208 falcen6-scan.a.pri falcen6-scan
#192.168.100.209 falcen6-scan.a.pri falcen6-scan
#192.168.100.210 falcen6-scan.a.pri falcen6-scan
on RAC2
[oracle@falcen6b ~]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
#Public
192.168.100.218 falcen6a.a.pri falcen6a
192.168.100.219 falcen6b.a.pri falcen6b
# Private
192.168.210.101 falcen6a-priv.a.pri falcen6a-priv
192.168.210.102 falcen6b-priv.a.pri falcen6b-priv
# Virtual
192.168.100.212 falcen6a-vip.a.pri falcen6a-vip
192.168.100.213 falcen6b-vip.a.pri falcen6b-vip
# SCAN
#192.168.100.208 falcen6-scan.a.pri falcen6-scan
#192.168.100.209 falcen6-scan.a.pri falcen6-scan
#192.168.100.210 falcen6-scan.a.pri falcen6-scan
Can someone please confirm this??

Similar Messages

  • Root.sh fails on second node

    I already posted this issue on database installation forum, and was suggested to post it on this forum.
    Here are the details.
    I am running Linux 64bit on ESx clients. Installing Oracle 11gR2.
    It passed all the per-requisite. Run root.sh on first node. It finished with no errorrs.
    On second node I got the following:
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /u01/app/11.2.0/grid
    Enter the full pathname of the local bin directory: [usr/local/bin]:
    The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-07-13 12:51:28: Parsing the host name
    2010-07-13 12:51:28: Checking for super user privileges
    2010-07-13 12:51:28: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node fred0224, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'fred0225'
    CRS-2676: Start of 'ora.mdnsd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'fred0225'
    CRS-2676: Start of 'ora.gipcd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'fred0225'
    CRS-2676: Start of 'ora.gpnpd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'fred0225'
    CRS-2676: Start of 'ora.cssdmonitor' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'fred0225'
    CRS-2672: Attempting to start 'ora.diskmon' on 'fred0225'
    CRS-2676: Start of 'ora.diskmon' on 'fred0225' succeeded
    CRS-2676: Start of 'ora.cssd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'fred0225'
    Start action for octssd aborted
    CRS-2676: Start of 'ora.ctssd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'fred0225'
    CRS-2672: Attempting to start 'ora.asm' on 'fred0225'
    CRS-2676: Start of 'ora.drivers.acfs' on 'fred0225' succeeded
    CRS-2676: Start of 'ora.asm' on 'fred0225' succeeded
    CRS-2664: Resource 'ora.ctssd' is already running on 'fred0225'
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.asm -init
    Start of resource "ora.asm -init" failed
    Failed to start ASM
    Failed to start Oracle Clusterware stack
    In the ocssd.log I found
    [ CSSD][3559689984]clssnmvDHBValidateNCopy: node 1, fred0224, has a disk HB, but no network HB, DHB has rcfg 174483948, wrtcnt, 232, LATS 521702664, lastSeqNo 232, uniqueness 1279039649, timestamp 1279039959/521874274
    In oraagent_oracle.log I found
    [ clsdmc][1212365120]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD)) with status 9
    2010-07-13 12:54:07.234: [ora.gpnpd][1212365120] [check] Error = error 9 encountered when connecting to GPNPD
    2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Calling PID check for daemon
    2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Trying to check PID = 20584
    2010-07-13 12:54:07.432: [ COMMCRS][1285794112]clsc_connect: (0x1304d850) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD))
    [ clsdmc][1222854976]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD)) with status 9
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Error = error 9 encountered when connecting to MDNSD
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Calling PID check for daemon
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Trying to check PID = 20571
    2010-07-13 12:54:08.841: [ COMMCRS][1201875264]clsc_connect: (0x12f3b1d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD))
    [ clsdmc][1159915840]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD)) with status 9
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Error = error 9 encountered when connecting to GIPCD
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Calling PID check for daemon
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Trying to check PID = 20566
    2010-07-13 12:54:10.242: [ COMMCRS][1254324544]clsc_connect: (0x12f35630) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD))
    In oracssdagent_root.log I found
    2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clssscConnect: gipc request failed with 29 (0x16)
    2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clsssInitNative: connect failed, rc 29
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clssnsqlnum: RPC failed rc 3
    2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_cssini: failed 3 to fetch node number
    2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_init: css init done, nodenum -1.
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssRecvMsg: got a disconnect from the server while waiting for message type 43
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssGetNLSData: Failure receiving a msg, rc 3
    If you need more info, let me know.

    Well, the error clearly indicates that a communication problem exists on the private interconnect.
    Could this be a setting in ESX, which prevents some communication between the clients on the second network card? Any routing table in ESX not configured correctly?
    Sebastian

  • Root.sh failed on second node while installing CRS 10g on centos 5.5

    root.sh failed on second node while installing CRS 10g
    Hi all,
    I am able to install Oracle 10g RAC clusterware on first node of the cluster. However, when I run the root.sh script as root
    user on second node of the cluster, it fails with following error message:
    NO KEYS WERE WRITTEN. Supply -force parameter to override.
    -force is destructive and will destroy any previous cluster
    configuration.
    Oracle Cluster Registry for cluster has already been initialized
    Startup will be queued to init within 90 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Failure at final check of Oracle CRS stack.
    10
    and run cluvfy stage -post hwos -n all -verbose,it show message:
    ERROR:
    Could not find a suitable set of interfaces for VIPs.
    Result: Node connectivity check failed.
    Checking shared storage accessibility...
    Disk Sharing Nodes (2 in count)
    /dev/sda db2 db1
    and run cluvfy stage -pre crsinst -n all -verbose,it show message:
    ERROR:
    Could not find a suitable set of interfaces for VIPs.
    Result: Node connectivity check failed.
    Checking system requirements for 'crs'...
    No checks registered for this product.
    and run cluvfy stage -post crsinst -n all -verbose,it show message:
    Result: Node reachability check passed from node "DB2".
    Result: User equivalence check passed for user "oracle".
    Node Name CRS daemon CSS daemon EVM daemon
    db2 no no no
    db1 yes yes yes
    Check: Health of CRS
    Node Name CRS OK?
    db1 unknown
    Result: CRS health check failed.
    check crsd.log and show message:
    clsc_connect: (0x143ca610) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_db2_crs))
    clsssInitNative: connect failed, rc 9
    Any help would be greatly appreciated.
    Edited by: 868121 on 2011-6-24 上午12:31

    Hello, it took a little searching, but I found this in a note in the GRID installation guide for Linux/UNIX:
    Public IP addresses and virtual IP addresses must be in the same subnet.
    In your case, you are using two different subnets for the VIPs.

  • Root.sh failed at second node OUL 6.3 Oracle GRID 11.2.0.3

    Hi, im installing a two node cluster mounted on Oracle Linux 6.3 with Oracle DB 11.2.0.3, the installation went smooth up until the execution of the root.sh script on the second node.
    THe script return this final lines:
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node nodo1, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    Start of resource "ora.crsd" failed
    CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'nodo2'
    CRS-4000: Command Start failed, or completed with errors.
    Failed to start Oracle Grid Infrastructure stack
    Failed to start Cluster Ready Services at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1286.
    /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
    In $GRID_HOME/log/node2/alertnode.log It appears to be a Cluster Time Synchronization Service issue, (i didn't synchronyze the nodes..) however the CTSS is running in observer mode, wich i believe it shouldn't affect the installation process. After that i lost it...there's an entry CRS-5018 indicating that an unused HAIP route was removed... and then, out of the blue: CRS-5818:Aborted command 'start' for resource 'ora.asm'. Some clarification will be deeply apreciated.
    Here's the complete log:
    2013-04-01 13:39:35.358
    [client(12163)]CRS-2101:The OLR was formatted using version 3.
    2013-04-01 19:40:19.597
    [ohasd(12338)]CRS-2112:The OLR service started on node nodo2.
    2013-04-01 19:40:19.657
    [ohasd(12338)]CRS-1301:Oracle High Availability Service started on node nodo2.
    [client(12526)]CRS-10001:01-Apr-13 13:41 ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.39-400.17.2.el6uek.i686'
    [client(12528)]CRS-10001:01-Apr-13 13:41 ACFS-9201: Not Supported
    [client(12603)]CRS-10001:01-Apr-13 13:41 ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.39-400.17.2.el6uek.i686'
    2013-04-01 19:41:17.509
    [ohasd(12338)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2013-04-01 19:41:17.618
    [gpnpd(12695)]CRS-2328:GPNPD started on node nodo2.
    2013-04-01 19:41:21.363
    [cssd(12755)]CRS-1713:CSSD daemon is started in exclusive mode
    2013-04-01 19:41:23.194
    [ohasd(12338)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
    2013-04-01 19:41:56.144
    [cssd(12755)]CRS-1707:Lease acquisition for node nodo2 number 2 completed
    2013-04-01 19:41:57.545
    [cssd(12755)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/ASM_DISK_1; details in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log.
    [cssd(12755)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node nodo1 and is terminating; details at (:CSSNM00006:) in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log
    2013-04-01 19:41:58.549
    [ohasd(12338)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'nodo2'.
    2013-04-01 19:42:10.025
    [gpnpd(12695)]CRS-2329:GPNPD on node nodo2 shutdown.
    2013-04-01 19:42:11.407
    [mdnsd(12685)]CRS-5602:mDNS service stopping by request.
    2013-04-01 19:42:29.642
    [gpnpd(12947)]CRS-2328:GPNPD started on node nodo2.
    2013-04-01 19:42:33.241
    [cssd(13012)]CRS-1713:CSSD daemon is started in clustered mode
    2013-04-01 19:42:35.104
    [ohasd(12338)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
    2013-04-01 19:42:44.065
    [cssd(13012)]CRS-1707:Lease acquisition for node nodo2 number 2 completed
    2013-04-01 19:42:45.484
    [cssd(13012)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/ASM_DISK_1; details in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log.
    2013-04-01 19:42:52.138
    [cssd(13012)]CRS-1601:CSSD Reconfiguration complete. Active nodes are nodo1 nodo2 .
    2013-04-01 19:42:55.081
    [ctssd(13076)]CRS-2403:The Cluster Time Synchronization Service on host nodo2 is in observer mode.
    2013-04-01 19:42:55.581
    [ctssd(13076)]CRS-2401:The Cluster Time Synchronization Service started on host nodo2.
    2013-04-01 19:42:55.581
    [ctssd(13076)]CRS-2407:The new Cluster Time Synchronization Service reference node is host nodo1.
    2013-04-01 19:43:08.875
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 19:43:08.876
    [ctssd(13076)]CRS-2409:The clock on host nodo2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2013-04-01 19:43:13.565
    [u01/app/11.2.0/grid/bin/orarootagent.bin(13064)]CRS-5018:(:CLSN00037:) Removed unused HAIP route: 169.254.0.0 / 255.255.0.0 / 0.0.0.0 / eth0
    2013-04-01 19:53:09.800
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5818:Aborted command 'start' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:0:223} in /u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log.
    2013-04-01 19:53:11.827
    [ohasd(12338)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) {0:0:223} in /u01/app/11.2.0/grid/log/nodo2/ohasd/ohasd.log.
    2013-04-01 19:53:12.779
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:53:13.892
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:53:43.877
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:54:13.891
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:54:43.906
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:55:13.914
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:55:43.918
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:56:13.922
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:56:53.209
    [crsd(13741)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:07:01.128
    [crsd(13741)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:07:01.278
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:07:08.689
    [crsd(15248)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:13:10.138
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 20:17:13.024
    [crsd(15248)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:17:13.171
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:17:20.826
    [crsd(16746)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:27:25.020
    [crsd(16746)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:27:25.176
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:27:31.591
    [crsd(18266)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:37:35.668
    [crsd(18266)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:37:35.808
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:37:43.209
    [crsd(19762)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:43:11.160
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 20:47:47.487
    [crsd(19762)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:47:47.637
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:47:55.086
    [crsd(21242)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:57:59.343
    [crsd(21242)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:57:59.492
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:58:06.996
    [crsd(22744)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:08:11.046
    [crsd(22744)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:08:11.192
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:08:18.726
    [crsd(24260)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:13:12.000
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 21:18:22.262
    [crsd(24260)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:18:22.411
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:18:29.927
    [crsd(25759)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:28:34.467
    [crsd(25759)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:28:34.616
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:28:41.990
    [crsd(27291)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:38:45.012
    [crsd(27291)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:38:45.160
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:38:52.790
    [crsd(28784)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:43:12.378
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 21:48:56.285
    [crsd(28784)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:48:56.435
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:49:04.421
    [crsd(30272)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:59:08.183
    [crsd(30272)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:59:08.318
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:59:15.860
    [crsd(31772)]CRS-1012:The OCR service started on node nodo2.

    Hi santysharma, thanks for the reply, i have two ethernet interfaces: eth0 (public network 192.168.1.0) and eth1 (private network 10.5.3.0), there is no device using that ip range, here's the output of route command:
    (Sorry for the alignment, i tried to tab it but the editor trims it again)
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    default 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
    private * 255.255.255.0 U 0 0 0 eth1
    link-local * 255.255.0.0 U 1002 0 0 eth0
    link-local * 255.255.0.0 U 1003 0 0 eth1
    public * 255.255.255.0 U 0 0 0 eth0
    And the /etc/hosts file
    127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
    10.5.3.1 nodo1.cluster nodo1
    10.5.3.2 nodo2.cluster nodo2
    192.168.1.13 cluster-scan
    192.168.1.14 nodo1-vip
    192.168.1.15 nodo2-vip
    And the ifconfig -a
    eth0 Link encap:Ethernet HWaddr C8:3A:35:D9:C6:2B
    inet addr:192.168.1.12 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::ca3a:35ff:fed9:c62b/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:34708 errors:0 dropped:18 overruns:0 frame:0
    TX packets:24693 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:48545969 (46.2 MiB) TX bytes:1994381 (1.9 MiB)
    eth1 Link encap:Ethernet HWaddr 00:0D:87:D0:A3:8E
    inet addr:10.5.3.2 Bcast:10.5.3.255 Mask:255.255.255.0
    inet6 addr: fe80::20d:87ff:fed0:a38e/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:0 (0.0 b) TX bytes:5344 (5.2 KiB)
    Interrupt:23 Base address:0x6000
    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:20 errors:0 dropped:0 overruns:0 frame:0
    TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:1320 (1.2 KiB) TX bytes:1320 (1.2 KiB)
    Now that i'm thinking i've read somewhere that ipv6 was no supported...yet there's no relation with the 169.254.x.x ip range.

  • ASM install fails on one node

    I have been trying to install 10gRAC on a two virtual node cluster. I installed clusterware and it was successful. Before I started ASM install:
    [oracle@rac1 bin]$ ./crs_stat -t
    Name Type Target State Host
    ora.rac1.gsd application ONLINE ONLINE rac1
    ora.rac1.ons application ONLINE ONLINE rac1
    ora.rac1.vip application ONLINE ONLINE rac1
    ora.rac2.gsd application ONLINE ONLINE rac2
    ora.rac2.ons application ONLINE ONLINE rac2
    ora.rac2.vip application ONLINE ONLINE rac2
    [oracle@rac1 logs]$ /u01/crs/oracle/product/10.2.0/crs/bin/crsctl check crs
    CSS appears healthy
    CRS appears healthy
    EVM appears healthy
    [oracle@rac1 logs]$ ps -ef|grep d.bin
    root 3795 1 0 13:57 ? 00:00:35 /u01/crs/oracle/product/10.2.0/crs/bin/crsd.bin reboot
    oracle 4966 3793 0 13:59 ? 00:00:06 /u01/crs/oracle/product/10.2.0/crs/bin/evmd.bin
    oracle 5082 5059 0 13:59 ? 00:01:06 /u01/crs/oracle/product/10.2.0/crs/bin/ocssd.bin
    oracle 30520 4813 0 16:23 pts/3 00:00:00 grep d.bin
    During thhe ASM install here is what I got:
    WARNING: Error while copying directory /u01/app/oracle/product/10.2.0/db_1 with exclude file list 'null' to nodes 'rac2'. [PRKC-1073 : Failed to transfer directory "/u01/app/oracle/product/10.2.0/db_1" to any of the given nodes "rac2 ".
    Error on node rac2:Read from remote host rac2: Connection reset by peer]
    Refer to '/u01/app/oracle/oraInventory/logs/installActions2009-04-18_01-30-26PM.log' for details. You may fix the errors on the required remote nodes. Refer to the install guide for error recovery. Click 'Yes' if you want to proceed. Click 'No' to exit the install. Do you want to continue?
    INFO: User Selected: Yes/OK
    It appears to me as though the installer was not able to copy over the "/u01/app/oracle/product/10.2.0/db_1" directory to the rac2 node. I do not see any reason for that, I have setup ssh user equivalence for both oracle and root users, ssh and scp seem to work both ways. Permissions should not be an issue on one node and not the other as I replicated the permissions.
    I continued the installation and ASM is working fine on rac1 node and not on the second node. I tried using the dbca to setup the ASM on the second node and it errors out with a "crs-0223 resource placement error". Here is what I did next:
    [oracle@rac1 bin]$ ./srvctl status asm -n rac1
    ASM instance +ASM1 is running on node rac1.
    [oracle@rac1 bin]$ ./srvctl status asm -n rac2
    ASM instance +ASM2 is not running on node rac2.
    [oracle@rac1 bin]$ ./crs_sta
    crs_start crs_start.bin crs_stat crs_stat.bin
    [oracle@rac1 bin]$ ./crs_stat -t
    Name Type Target State Host
    ora....SM1.asm application ONLINE ONLINE rac1
    ora....C1.lsnr application ONLINE ONLINE rac1
    ora.rac1.gsd application ONLINE ONLINE rac1
    ora.rac1.ons application ONLINE ONLINE rac1
    ora.rac1.vip application ONLINE ONLINE rac1
    ora....SM2.asm application ONLINE UNKNOWN rac2
    ora....C2.lsnr application ONLINE UNKNOWN rac2
    ora.rac2.gsd application ONLINE ONLINE rac2
    ora.rac2.ons application ONLINE ONLINE rac2
    ora.rac2.vip application ONLINE ONLINE rac2
    [oracle@rac1 bin]$ ./crs_start ora.rac2.ASM2.asm
    CRS-1028: Dependency analysis failed because of:
    'Resource in UNKNOWN state: ora.rac2.ASM2.asm'
    CRS-0223: Resource 'ora.rac2.ASM2.asm' has placement error.
    I would like to get the ASM instance extended to the second node (rac2) and ofcourse, continue with the database instance creation. How can I accomplish this?
    Thanks!

    Hi orafun,
    this message:
    +WARNING: Error while copying directory /u01/app/oracle/product/10.2.0/db_1 with exclude file list 'null' to nodes 'rac2'. [PRKC-1073 : Failed to transfer directory "/u01/app/oracle/product/10.2.0/db_1" to any of the given nodes "rac2 ".+
    +Error on node rac2:Read from remote host rac2: Connection reset by peer]+
    Refer to '/u01/app/oracle/oraInventory/logs/installActions2009-04-18_01-30-26PM.log' for details. You may fix the errors on the required remote nodes. Refer to the install guide for error recovery. Click 'Yes' if you want to proceed. Click 'No' to exit the install. Do you want to continue?
    INFO: User Selected: Yes/OK
    Tells you that the Oracle Home could not be copied onto the remote node. The logs mentioned might tell you more, but this is the reason why ASM cannot be started on the other node - there is no software that could be used to start an ASM instance. Now you said:
    "+It appears to me as though the installer was not able to copy over the "/u01/app/oracle/product/10.2.0/db_1" directory to the rac2 node. I do not see any reason for that, I have setup ssh user equivalence for both oracle and root users, ssh and scp seem to work both ways. Permissions should not be an issue on one node and not the other as I replicated the permissions+."
    My question would be: What do you try to achieve? IF it is your only interest to "get it done and over with", then you can TAR up the Oracle Database home from which you want to run ASM and un-TAR on the remote node. Given that the paths are all correct, the registration already took place and hence, you can try starting the ASM instance on node2. IF you want to know the reason for the issue, further investigation and more information would be required.
    Hope that helps. Thanks,
    Markus

  • Root.sh fails on second node during clusterware installation

    I am setting up a test instance of OEL 5.4 using VMware.
    I am running the clusterware install and it is failing only on node2. See below.
    I followed note 414897.1 on metalink for raw device setup.
    Any help would be greatly appreciate.
    2010-09-01 11:58:21.084: [ default][1275584]a_init:7!: Backend init unsuccessful : [22]
    2010-09-01 11:58:21.091: [  OCRRAW][1275584]propriogid:1: INVALID FORMAT
    2010-09-01 11:58:21.091: [  OCRRAW][1275584]ibctx:1:ERROR: INVALID FORMAT
    2010-09-01 11:58:21.091: [  OCRRAW][1275584]proprinit:problem reading the bootblock or superbloc 22
    2010-09-01 11:58:21.097: [  OCRRAW][1275584]propriogid:1: INVALID FORMAT
    2010-09-01 11:58:21.139: [  OCRRAW][1275584]propriowv: Vote information on disk 0 [u01/app/oracle/oradata/ocr] is adjusted from [0/0] to [2/2]
    2010-09-01 11:58:21.191: [  OCRRAW][1275584]propriniconfig:No 92 configuration
    2010-09-01 11:58:21.192: [  OCRAPI][1275584]a_init:6a: Backend init successful
    2010-09-01 11:58:21.299: [ OCRCONF][1275584]Initialized DATABASE keys in OCR
    2010-09-01 11:58:21.555: [ OCRCONF][1275584]Successfully set skgfr block 0
    2010-09-01 11:58:21.557: [ OCRCONF][1275584]Exiting [status=success]...

    Oracle 10gR2 RAC Installation in RedHat 5 Linux Using VMware.
    Important points to install 10gR2 oracle RAC in linux5.
    1.LINUX 5(Redhat 5) doesn't have /etc/sysconfig/rawdevices file. so we have to configure it.
    2. Edit the /etc/redhat-release version to redhat-4 and and to invoke the runInstaller use the command
    $runInstaller -ignoreSysPrereqs. //this will bypass the os check //
    3. Next during clusterware installation at the end of root.sh in node 2 end with error message.So we have adjust the parameters in vipca and srvctl files.
    4. vipca will fail to run. so we have to adjust some parameters and configure it manually.
    refer the link, it will be useful to you to complete your installation.
    http://oracleinstance.blogspot.com/2010/03/oracle-10g-installation-in-linux-5.html

  • The Script root.sh problem - ora.asm and ASM and Clusterware Stack failed

    Folks,
    Hello. I am installing Oracle 11gR2 RAC using 2 VMs (rac1 and rac2) whose OS are Oracle Linux 5.6 in VMPlayer according to the website http://appsdbaworkshop.blogspot.com/2011/10/11gr2-rac-on-linux-56-using-vmware.html
    I am installing Grid infrastructure. On step 9 of 10 - execute script /u01/app/grid/root.sh for 2 VMs rac1 and rac2.
    After run root.sh in rac1 successfully. I run root.sh in rac2 and get an error as below:
    [root@rac2 grid]# ./root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
    ORACLE_OWNER= ora11g
    ORACLE_HOME= /u01/app/grid
    Enter the full pathname of the local bin directory: [usr/local/bin]: /usr/local/bin
    Copying dbhome to /usr/local/bin ...
    Copying oraenv to /usr/local/bin ...
    Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2012-03-05 16:32:52: Parsing the host name
    2012-03-05 16:32:52: Checking for super user privileges
    2012-03-05 16:32:52: User has super user privileges
    Using configuration parameter file: /u01/app/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
    Start action for octssd aborted
    CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac2'
    CRS-2672: Attempting to start 'ora.asm' on 'rac2'
    CRS-2676: Start of 'ora.drivers.acfs' on 'rac2' succeeded
    CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
    CRS-2664: Resource 'ora.ctssd' is already running on 'rac2'
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/app/grid/bin/crsctl start resource ora.asm -init
    Start of resource "ora.asm -init" failed
    Failed to start ASM
    Failed to start Oracle Clusterware stack
    [root@rac2 grid]#
    As we see the output above, at the end of the output
    1) Start of resource ora.asm -init failed
    2) Failed to start ASM
    3) Failed to start Oracle Clusterware stack
    The runInstaller is in the first VM rac1. My question is:
    Do any folk understand how to solve the script root.sh in rac2 problem ( 3 fails of ora.asm, ASM and Clusterware stack as above) ?
    Thanks.

    Please check there is no firewall exist:
    try this like:
    root.sh fails on second node
    MOS note:
    11gR2 Grid: root.sh Fails to Start the Clusterware on the Second Node Due to Firewall on Private Network [ID 981357.1]
    Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1] (Most probabily this issue)

  • Root.sh on second node fails

    I am running Linux 64bit. Installing Oracle 11gR2.
    It passed all the per-requisite. Run root.sh on first node. It finished with no errorrs.
    On second node I got the following:
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /u01/app/11.2.0/grid
    Enter the full pathname of the local bin directory: [usr/local/bin]:
    The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-07-13 12:51:28: Parsing the host name
    2010-07-13 12:51:28: Checking for super user privileges
    2010-07-13 12:51:28: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node fred0224, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'fred0225'
    CRS-2676: Start of 'ora.mdnsd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'fred0225'
    CRS-2676: Start of 'ora.gipcd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'fred0225'
    CRS-2676: Start of 'ora.gpnpd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'fred0225'
    CRS-2676: Start of 'ora.cssdmonitor' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'fred0225'
    CRS-2672: Attempting to start 'ora.diskmon' on 'fred0225'
    CRS-2676: Start of 'ora.diskmon' on 'fred0225' succeeded
    CRS-2676: Start of 'ora.cssd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'fred0225'
    Start action for octssd aborted
    CRS-2676: Start of 'ora.ctssd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'fred0225'
    CRS-2672: Attempting to start 'ora.asm' on 'fred0225'
    CRS-2676: Start of 'ora.drivers.acfs' on 'fred0225' succeeded
    CRS-2676: Start of 'ora.asm' on 'fred0225' succeeded
    CRS-2664: Resource 'ora.ctssd' is already running on 'fred0225'
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.asm -init
    Start of resource "ora.asm -init" failed
    Failed to start ASM
    Failed to start Oracle Clusterware stack
    In the ocssd.log I found
    [    CSSD][3559689984]clssnmvDHBValidateNCopy: node 1, fred0224, has a disk HB, but no network HB, DHB has rcfg 174483948, wrtcnt, 232, LATS 521702664, lastSeqNo 232, uniqueness 1279039649, timestamp 1279039959/521874274
    In oraagent_oracle.log I found
    [  clsdmc][1212365120]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD)) with status 9
    2010-07-13 12:54:07.234: [ora.gpnpd][1212365120] [check] Error = error 9 encountered when connecting to GPNPD
    2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Calling PID check for daemon
    2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Trying to check PID = 20584
    2010-07-13 12:54:07.432: [ COMMCRS][1285794112]clsc_connect: (0x1304d850) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD))
    [  clsdmc][1222854976]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD)) with status 9
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Error = error 9 encountered when connecting to MDNSD
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Calling PID check for daemon
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Trying to check PID = 20571
    2010-07-13 12:54:08.841: [ COMMCRS][1201875264]clsc_connect: (0x12f3b1d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD))
    [  clsdmc][1159915840]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD)) with status 9
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Error = error 9 encountered when connecting to GIPCD
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Calling PID check for daemon
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Trying to check PID = 20566
    2010-07-13 12:54:10.242: [ COMMCRS][1254324544]clsc_connect: (0x12f35630) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD))
    In oracssdagent_root.log I found
    2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clssscConnect: gipc request failed with 29 (0x16)
    2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clsssInitNative: connect failed, rc 29
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clssnsqlnum: RPC failed rc 3
    2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_cssini: failed 3 to fetch node number
    2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_init: css init done, nodenum -1.
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssRecvMsg: got a disconnect from the server while waiting for message type 43
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssGetNLSData: Failure receiving a msg, rc 3
    If anyone needs more info please let me know.

    On all nodes,
    1. Modify the /etc/sysconfig/oracleasm with:
    ORACLEASM_SCANORDER="dm"
    ORACLEASM_SCANEXCLUDE="sd"
    2. restart the asmlib by :
    # /etc/init.d/oracleasm restart
    3. Run root.sh on the 2nd node
    hope this helps you

  • 11G R2 root.sh failed on first node with OLE fetch parameter error

    I have successfully installed 11G R2.1 on Centos 5.4 64 bit.
    Now it's coming to install 11G R2.2 on Redhat 5.4 64bit with HDS storrage.
    [grid@dmdb1 grid]$ uname -a
    Linux dmdb1 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
    I passed all pre-ins requirements except shared storage. However, I manually verify it with no problems.
    [grid@dmdb1 grid]$ ./runcluvfy.sh stage -pre crsinst -fixup -n dmdb1,dmdb2,dmdb3,dmdb4 -verbose|grep -i fail
    [grid@dmdb1 grid]$ ./runcluvfy.sh stage -post hwos -n dmdb1,dmdb2,dmdb3,dmdb4 -verbose|grep -i fail
    [grid@dmdb1 grid]$ ./runcluvfy.sh comp sys -n dmdb1,dmdb2,dmdb3,dmdb4 -p crs -osdba dba -orainv oinstall
    Verifying system requirement
    Total memory check passed
    Available memory check passed
    Swap space check passed
    Free disk space check passed for "dmdb4:/tmp"
    Free disk space check passed for "dmdb3:/tmp"
    Free disk space check passed for "dmdb2:/tmp"
    Free disk space check passed for "dmdb1:/tmp"
    User existence check passed for "grid"
    Group existence check passed for "oinstall"
    Group existence check passed for "dba"
    Membership check for user "grid" in group "oinstall" [as Primary] passed
    Membership check for user "grid" in group "dba" passed
    Run level check passed
    Hard limits check passed for "maximum open file descriptors"
    Soft limits check passed for "maximum open file descriptors"
    Hard limits check passed for "maximum user processes"
    Soft limits check passed for "maximum user processes"
    System architecture check passed
    Kernel version check passed
    Kernel parameter check passed for "semmsl"
    Kernel parameter check passed for "semmns"
    Kernel parameter check passed for "semopm"
    Kernel parameter check passed for "semmni"
    Kernel parameter check passed for "shmmax"
    Kernel parameter check passed for "shmmni"
    Kernel parameter check passed for "shmall"
    Kernel parameter check passed for "file-max"
    Kernel parameter check passed for "ip_local_port_range"
    Kernel parameter check passed for "rmem_default"
    Kernel parameter check passed for "rmem_max"
    Kernel parameter check passed for "wmem_default"
    Kernel parameter check passed for "wmem_max"
    Kernel parameter check passed for "aio-max-nr"
    Package existence check passed for "make-3.81"
    Package existence check passed for "binutils-2.17.50.0.6"
    Package existence check passed for "gcc-4.1"
    Package existence check passed for "libaio-0.3.106 (i386)"
    Package existence check passed for "libaio-0.3.106 (x86_64)"
    Package existence check passed for "glibc-2.5-24 (i686)"
    Package existence check passed for "glibc-2.5-24 (x86_64)"
    Package existence check passed for "compat-libstdc++-33-3.2.3 (i386)"
    Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)"
    Package existence check passed for "elfutils-libelf-0.125 (x86_64)"
    Package existence check passed for "elfutils-libelf-devel-0.125"
    Package existence check passed for "glibc-common-2.5"
    Package existence check passed for "glibc-devel-2.5 (i386)"
    Package existence check passed for "glibc-devel-2.5 (x86_64)"
    Package existence check passed for "glibc-headers-2.5"
    Package existence check passed for "gcc-c++-4.1.2"
    Package existence check passed for "libaio-devel-0.3.106 (i386)"
    Package existence check passed for "libaio-devel-0.3.106 (x86_64)"
    Package existence check passed for "libgcc-4.1.2 (i386)"
    Package existence check passed for "libgcc-4.1.2 (x86_64)"
    Package existence check passed for "libstdc++-4.1.2 (i386)"
    Package existence check passed for "libstdc++-4.1.2 (x86_64)"
    Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)"
    Package existence check passed for "sysstat-7.0.2"
    Package existence check passed for "unixODBC-2.2.11 (i386)"
    Package existence check passed for "unixODBC-2.2.11 (x86_64)"
    Package existence check passed for "unixODBC-devel-2.2.11 (i386)"
    Package existence check passed for "unixODBC-devel-2.2.11 (x86_64)"
    Package existence check passed for "ksh-20060214"
    Check for multiple users with UID value 0 passed
    Verification of system requirement was successful.
    [grid@dmdb1 grid]$ ./runcluvfy.sh comp sys -n dmdb1,dmdb2,dmdb3,dmdb4 -p database -osdba dba -orainv oinstall|grep -i fail
    [grid@dmdb1 grid]$ ./runcluvfy.sh comp ssa -n dmdb1,dmdb2,dmdb3,dmdb4
    Verifying shared storage accessibility
    Checking shared storage accessibility...
    Storage operation failed
    Shared storage check failed on nodes "dmdb4,dmdb3,dmdb2,dmdb1"
    Verification of shared storage accessibility was unsuccessful on all the specified nodes.
    I followed below article to verify shared storage issues:
    http://www.webofwood.com/rac/oracle-response-to-shared-storage-check-failed-on-nodes/
    it's ok.
    So I skipped SSA issue and go on install with (./runInstaller -ignoreInternalDriverError).
    However, when I ran root.sh with below error:
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'dmdb1'
    CRS-2677: Stop of 'ora.mdnsd' on 'dmdb1' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'dmdb1'
    CRS-2677: Stop of 'ora.gipcd' on 'dmdb1' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    CRS-2672: Attempting to start 'ora.gipcd' on 'dmdb1'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'dmdb1'
    CRS-2676: Start of 'ora.gipcd' on 'dmdb1' succeeded
    CRS-2676: Start of 'ora.mdnsd' on 'dmdb1' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'dmdb1'
    CRS-2676: Start of 'ora.gpnpd' on 'dmdb1' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dmdb1'
    CRS-2676: Start of 'ora.cssdmonitor' on 'dmdb1' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'dmdb1'
    CRS-2672: Attempting to start 'ora.diskmon' on 'dmdb1'
    CRS-2676: Start of 'ora.diskmon' on 'dmdb1' succeeded
    CRS-2674: Start of 'ora.cssd' on 'dmdb1' failed
    CRS-2679: Attempting to clean 'ora.cssd' on 'dmdb1'
    CRS-2681: Clean of 'ora.cssd' on 'dmdb1' succeeded
    CRS-2673: Attempting to stop 'ora.diskmon' on 'dmdb1'
    CRS-2677: Stop of 'ora.diskmon' on 'dmdb1' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'dmdb1'
    CRS-2677: Stop of 'ora.gpnpd' on 'dmdb1' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'dmdb1'
    CRS-2677: Stop of 'ora.mdnsd' on 'dmdb1' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'dmdb1'
    CRS-2677: Stop of 'ora.gipcd' on 'dmdb1' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /opt/app/11.2.0/grid/bin/crsctl start resource ora.ctssd -init
    Start of resource "ora.ctssd -init" failed
    Clusterware exclusive mode start of resource ora.ctssd failed
    CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /opt/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
    Stop of resource "ora.crsd -init" failed
    Failed to stop CRSD
    CRS-2500: Cannot stop resource 'ora.asm' as it is not running
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /opt/app/11.2.0/grid/bin/crsctl stop resource ora.asm -init
    Stop of resource "ora.asm -init" failed
    Failed to stop ASM
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'dmdb1'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'dmdb1' succeeded
    Initial cluster configuration failed. See /opt/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_dmdb1.log for details
    I manually ran '/opt/app/11.2.0/grid/bin/crsctl start resource ora.ctssd -init' and got below erros from /opt/app/11.2.0/grid/log/dmdb1/cssd/ocssd.log
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2011-09-23 19:06:41.501: [    CSSD][1812336384]clssscmain: Starting CSS daemon, version 11.2.0.1.0, in (exclusive) mode with uniqueness value 1316776001
    2011-09-23 19:06:41.502: [    CSSD][1812336384]clssscmain: Environment is production
    2011-09-23 19:06:41.502: [    CSSD][1812336384]clssscmain: Core file size limit extended
    2011-09-23 19:06:41.515: [    CSSD][1812336384]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
    2011-09-23 19:06:41.515: [    CSSD][1812336384]clssscSetPrivEnv: IPMI device not installed on this node
    2011-09-23 19:06:41.517: [    CSSD][1812336384]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
    2011-09-23 19:06:41.539: [    CSSD][1812336384]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
    2011-09-23 19:06:41.539: [    CSSD][1812336384]clssscExtendLimits: The current soft limit for locked memory is 4294967295, hard limit is 4294967295
    2011-09-23 19:06:41.541: [    CSSD][1812336384]clssscmain: Running as user grid
    anybody can help me fix it?

    I opened on SR for this case.
    it's ok now.
    Below is from Oracle Global Service request:
    === ODM Action Plan ===
    Dear customer, after went through the uploaded log files, we found the issue looks like
    bug 9732641 : The clusterware gpnpd process crashes when there is more than 1 cluster with the same name.
    To narrow down the issue, pls apply the following steps.
    1. Pls clean the previous configuration with below steps, then run root.sh script on node1 again.
    1.1 remove current configuration.
    $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force
    1.2 remove other related files.
    if $GI_BASE/Clusterware/ckptGridHA_.xml still there, please remove it manually with "rm" command on all nodes
    If the gpnp profile is still there, pls clean up them, then rebuild require directories.
    $ rm -rf $GRID_HOME/gpnp/*
    $ mkdir -p $GRID_HOME/gpnp/profiles/peer $GRID_HOME/gpnp/wallets/peer $GRID_HOME/gpnp/wallets/prdr $GRID_HOME/gpnp/wallets/pa $GRID_HOME/gpnp/wallets/root
    2. After the previous configuration was cleaned up, pls rerun the root.sh script again. If the issue still there, pls upload the following:
    Everything under <GI_HOME>/log
    Everything under <ORACLE_BAES for grid user>/cfgtoollogs
    Everything under <GI_HOME>/cfgtolllogs/crsconfig
    OS log(/var/log/messages)
    3. Pls also make sure there is only one GI running on your cluster.
    See /opt/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_dmdb1.log for details

  • 11gR2: ASM configuration failed during root.sh execution.

    Hi All,
    Need your help, i am trying to install 11gR2 and using ASM for voting disk and ocr. when i executed root.sh then it failed saying "Initial cluster configuration failed" . when i checked log (opt/oracle/prod
    uct/CRS/cfgtoollogs/crsconfig/rootcrs_auto.log) then found
    2010-02-17 10:39:54: Configuring ASM via ASMCA
    2010-02-17 10:39:54: Executing as oracle: /opt/oracle/product/CRS/bin/asmca -sil
    ent -diskGroupName DATA -diskList /dev/vgora/rora_vote_disk,/dev/vgora/rora_ocr
    -redundancy EXTERNAL -diskString '/dev/vgora/*,/dev/vgasm/*,/dev/rdisk/*' -confi
    gureLocalASM
    2010-02-17 10:39:54: Running as user oracle: /opt/oracle/product/CRS/bin/asmca -
    silent -diskGroupName DATA -diskList /dev/vgora/rora_vote_disk,/dev/vgora/rora_o
    cr -redundancy EXTERNAL -diskString '/dev/vgora/*,/dev/vgasm/*,/dev/rdisk/*' -co
    nfigureLocalASM
    2010-02-17 10:39:54: Invoking "/opt/oracle/product/CRS/bin/asmca -silent -disk
    GroupName DATA -diskList /dev/vgora/rora_vote_disk,/dev/vgora/rora_ocr -redundan
    cy EXTERNAL -diskString '/dev/vgora/*,/dev/vgasm/*,/dev/rdisk/*' -configureLocal
    ASM" as user "oracle"
    2010-02-17 10:40:10: Configuration of ASM failed, see logs for details
    2010-02-17 10:40:10: Did not succssfully configure and start ASM
    2010-02-17 10:40:10: Exiting exclusive mode
    2010-02-17 10:40:10: Command return code of 1 (256) from command: /opt/oracle/pr
    oduct/CRS/bin/crsctl stop resource ora.crsd -init
    2010-02-17 10:40:10: Stop of resource "ora.crsd -init" failed
    2010-02-17 10:40:10: Failed to stop CRSD
    2010-02-17 10:40:11: Command return code of 1 (256) from command: /opt/oracle/pr
    oduct/CRS/bin/crsctl stop resource ora.asm -init
    2010-02-17 10:40:11: Stop of resource "ora.asm -init" failed
    2010-02-17 10:40:11: Failed to stop ASM
    2010-02-17 10:40:36: Initial cluster configuration failed. See /opt/oracle/prod
    uct/CRS/cfgtoollogs/crsconfig/rootcrs_auto.log for details
    Please guide me , how to proceed further.

    CSSD does not start (Cluster Manager)
    CSSD does not start (Cluster Manager)

  • Root.ah fails on 2nd node(rac2) with [ ORA-15018,ORA-15017,ORA-15003 ]

    Hi All,
    I m trying to setup 11gR2 Grid installation on two-node Rac . When it comes to running root.sh on second node (i.e. rac2) it fails with below error. Could please anyone help me out. This is my 3rd attempt and all fails with below errors on node 2.
    rac2:
    [root@rac2 grid_home]# ./root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= grid
        ORACLE_HOME=  /u01/grid_home
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
       Copying dbhome to /usr/local/bin ...
       Copying oraenv to /usr/local/bin ...
       Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2013-07-10 18:53:15: Parsing the host name
    2013-07-10 18:53:15: Checking for super user privileges
    2013-07-10 18:53:15: User has super user privileges
    Using configuration parameter file: /u01/grid_home/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
    CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
    DiskGroup CRS creation failed with the following message:
    ORA-15018: diskgroup cannot be created
    ORA-15017: diskgroup "CRS" cannot be mounted
    ORA-15003: diskgroup "CRS" already mounted in another lock name space
    Configuration of ASM failed, see logs for details
    Did not succssfully configure and start ASM
    CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/grid_home/bin/crsctl stop resource ora.crsd -init
    Stop of resource "ora.crsd -init" failed
    Failed to stop CRSD
    CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
    CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
    CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac2'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
    CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
    CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
    CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
    CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
    Initial cluster configuration failed.  See /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log for details
    [root@rac2 grid_home]#
    rac2  alertrac2.log
    [root@rac2 rac2]# cat -n alertrac2.log
         1  Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
         2  2013-07-10 18:53:16.145
         3  [client(13088)]CRS-2106:The OLR location /u01/grid_home/cdata/rac2.olr is inaccessible. Details in /u01/grid_home/log/rac2/client/ocrconfig_13088.log.
         4  2013-07-10 18:53:16.228
         5  [client(13088)]CRS-2101:The OLR was formatted using version 3.
         6  2013-07-10 18:53:31.734
         7  [ohasd(13132)]CRS-2112:The OLR service started on node rac2.
         8  2013-07-10 18:53:31.893
         9  [ohasd(13132)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
        10  2013-07-10 18:53:53.762
        11  [ohasd(13132)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
        12  2013-07-10 18:53:55.381
        13  [cssd(14409)]CRS-1713:CSSD daemon is started in exclusive mode
        14  2013-07-10 18:54:01.530
        15  [cssd(14409)]CRS-1709:Lease acquisition failed for node rac2 because no voting file has been configured; Details at (:CSSNM00031:) in /u01/grid_home/log/rac2/cssd/ocssd.log
        16  2013-07-10 18:54:19.113
        17  [cssd(14409)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
        18  2013-07-10 18:54:19.910
        19  [ctssd(14465)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
        20  2013-07-10 18:54:19.920
        21  [ctssd(14465)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
        22  2013-07-10 18:54:20.903
        23  [ctssd(14465)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
        24  [client(14715)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
        25  [client(14719)]CRS-10001:ACFS-9322: done.
        26  2013-07-10 18:54:47.104
        27  [ctssd(14465)]CRS-2405:The Cluster Time Synchronization Service on host rac2 is shutdown by user
        28  2013-07-10 18:54:55.837
        29  [cssd(14409)]CRS-1603:CSSD on node rac2 shutdown by user.
    rac2 rootcrs logfile
    [root@rac2 rac2]# cat  /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log
    2013-07-10 18:53:15: The configuration parameter file /u01/grid_home/crs/install/crsconfig_params is valid
    2013-07-10 18:53:15: Checking for super user privileges
    2013-07-10 18:53:15: User has super user privileges
    2013-07-10 18:53:15: ### Printing the configuration values from files:
    2013-07-10 18:53:15:    /u01/grid_home/crs/install/crsconfig_params
    2013-07-10 18:53:15:    /u01/grid_home/crs/install/s_crsconfig_defs
    2013-07-10 18:53:15: ASM_DISCOVERY_STRING=
    2013-07-10 18:53:15: ASM_DISKS=ORCL:CRS1
    2013-07-10 18:53:15: ASM_DISK_GROUP=CRS
    2013-07-10 18:53:15: ASM_REDUNDANCY=EXTERNAL
    2013-07-10 18:53:15: ASM_SPFILE=
    2013-07-10 18:53:15: ASM_UPGRADE=false
    2013-07-10 18:53:15: CLSCFG_MISSCOUNT=
    2013-07-10 18:53:15: CLUSTER_GUID=
    2013-07-10 18:53:15: CLUSTER_NAME=rac-scan
    2013-07-10 18:53:15: CRS_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'
    2013-07-10 18:53:15: CRS_STORAGE_OPTION=1
    2013-07-10 18:53:15: CSS_LEASEDURATION=400
    2013-07-10 18:53:15: DIRPREFIX=
    2013-07-10 18:53:15: DISABLE_OPROCD=0
    2013-07-10 18:53:15: EMBASEJAR_NAME=oemlt.jar
    2013-07-10 18:53:15: EWTJAR_NAME=ewt3.jar
    2013-07-10 18:53:15: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
    2013-07-10 18:53:15: GNS_ADDR_LIST=
    2013-07-10 18:53:15: GNS_ALLOW_NET_LIST=
    2013-07-10 18:53:15: GNS_CONF=false
    2013-07-10 18:53:15: GNS_DENY_ITF_LIST=
    2013-07-10 18:53:15: GNS_DENY_NET_LIST=
    2013-07-10 18:53:15: GNS_DOMAIN_LIST=
    2013-07-10 18:53:15: GPNPCONFIGDIR=/u01/grid_home
    2013-07-10 18:53:15: GPNPGCONFIGDIR=/u01/grid_home
    2013-07-10 18:53:15: GPNP_PA=
    2013-07-10 18:53:15: HELPJAR_NAME=help4.jar
    2013-07-10 18:53:15: HOST_NAME_LIST=rac1,rac2
    2013-07-10 18:53:15: ID=/etc/init.d
    2013-07-10 18:53:15: INIT=/sbin/init
    2013-07-10 18:53:15: IT=/etc/inittab
    2013-07-10 18:53:15: JEWTJAR_NAME=jewt4.jar
    2013-07-10 18:53:15: JLIBDIR=/u01/grid_home/jlib
    2013-07-10 18:53:15: JREDIR=/u01/grid_home/jdk/jre/
    2013-07-10 18:53:15: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
    2013-07-10 18:53:15: MSGFILE=/var/adm/messages
    2013-07-10 18:53:15: NETCFGJAR_NAME=netcfg.jar
    2013-07-10 18:53:15: NETWORKS="eth0"/192.168.0.0:public,"eth1"/192.168.1.0:cluster_interconnect
    2013-07-10 18:53:15: NEW_HOST_NAME_LIST=
    2013-07-10 18:53:15: NEW_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'
    2013-07-10 18:53:15: NEW_NODE_NAME_LIST=
    2013-07-10 18:53:15: NEW_PRIVATE_NAME_LIST=
    2013-07-10 18:53:15: NODELIST=rac1,rac2
    2013-07-10 18:53:15: NODE_NAME_LIST=rac1,rac2
    2013-07-10 18:53:15: OCFS_CONFIG=
    2013-07-10 18:53:15: OCRCONFIG=/etc/oracle/ocr.loc
    2013-07-10 18:53:15: OCRCONFIGDIR=/etc/oracle
    2013-07-10 18:53:15: OCRID=
    2013-07-10 18:53:15: OCRLOC=ocr.loc
    2013-07-10 18:53:15: OCR_LOCATIONS=NO_VAL
    2013-07-10 18:53:15: OLASTGASPDIR=/etc/oracle/lastgasp
    2013-07-10 18:53:15: OLRCONFIG=/etc/oracle/olr.loc
    2013-07-10 18:53:15: OLRCONFIGDIR=/etc/oracle
    2013-07-10 18:53:15: OLRLOC=olr.loc
    2013-07-10 18:53:15: OPROCDCHECKDIR=/etc/oracle/oprocd/check
    2013-07-10 18:53:15: OPROCDDIR=/etc/oracle/oprocd
    2013-07-10 18:53:15: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
    2013-07-10 18:53:15: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
    2013-07-10 18:53:15: ORACLE_BASE=/u01/11.2.0
    2013-07-10 18:53:15: ORACLE_HOME=/u01/grid_home
    2013-07-10 18:53:15: ORACLE_OWNER=grid
    2013-07-10 18:53:15: ORA_ASM_GROUP=asmadmin
    2013-07-10 18:53:15: ORA_DBA_GROUP=oinstall
    2013-07-10 18:53:15: PRIVATE_NAME_LIST=
    2013-07-10 18:53:15: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
    2013-07-10 18:53:15: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
    2013-07-10 18:53:15: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
    2013-07-10 18:53:15: RC_KILL=K19
    2013-07-10 18:53:15: RC_KILL_OLD=K96
    2013-07-10 18:53:15: RC_START=S96
    2013-07-10 18:53:15: SCAN_NAME=rac-scan.naveed.com
    2013-07-10 18:53:15: SCAN_PORT=1521
    2013-07-10 18:53:15: SCRBASE=/etc/oracle/scls_scr
    2013-07-10 18:53:15: SHAREJAR_NAME=share.jar
    2013-07-10 18:53:15: SILENT=false
    2013-07-10 18:53:15: SO_EXT=so
    2013-07-10 18:53:15: SRVCFGLOC=srvConfig.loc
    2013-07-10 18:53:15: SRVCONFIG=/var/opt/oracle/srvConfig.loc
    2013-07-10 18:53:15: SRVCONFIGDIR=/var/opt/oracle
    2013-07-10 18:53:15: VNDR_CLUSTER=false
    2013-07-10 18:53:15: VOTING_DISKS=NO_VAL
    2013-07-10 18:53:15: ### Printing other configuration values ###
    2013-07-10 18:53:15: CLSCFG_EXTRA_PARMS=
    2013-07-10 18:53:15: CRSDelete=0
    2013-07-10 18:53:15: CRSPatch=0
    2013-07-10 18:53:15: DEBUG=
    2013-07-10 18:53:15: DOWNGRADE=
    2013-07-10 18:53:15: HAS_GROUP=oinstall
    2013-07-10 18:53:15: HAS_USER=root
    2013-07-10 18:53:15: HOST=rac2
    2013-07-10 18:53:15: IS_SIHA=0
    2013-07-10 18:53:15: OLR_DIRECTORY=/u01/grid_home/cdata
    2013-07-10 18:53:15: OLR_LOCATION=/u01/grid_home/cdata/rac2.olr
    2013-07-10 18:53:15: ORA_CRS_HOME=/u01/grid_home
    2013-07-10 18:53:15: SUPERUSER=root
    2013-07-10 18:53:15: UPGRADE=
    2013-07-10 18:53:15: VF_DISCOVERY_STRING=
    2013-07-10 18:53:15: addfile=/u01/grid_home/crs/install/crsconfig_addparams
    2013-07-10 18:53:15: crscfg_trace=1
    2013-07-10 18:53:15: crscfg_trace_file=/u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log
    2013-07-10 18:53:15: hosts=
    2013-07-10 18:53:15: oldcrshome=
    2013-07-10 18:53:15: oldcrsver=
    2013-07-10 18:53:15: osdfile=/u01/grid_home/crs/install/s_crsconfig_defs
    2013-07-10 18:53:15: parameters_valid=1
    2013-07-10 18:53:15: paramfile=/u01/grid_home/crs/install/crsconfig_params
    2013-07-10 18:53:15: platform_family=unix
    2013-07-10 18:53:15: srvctl_trc_suff=0
    2013-07-10 18:53:15: unlock_crshome=
    2013-07-10 18:53:15: user_is_superuser=1
    2013-07-10 18:53:15: ### Printing of configuration values complete ###
    2013-07-10 18:53:15: Oracle CRS stack is not configured yet
    2013-07-10 18:53:15: CRS is not yet configured. Hence, will proceed to configure CRS
    2013-07-10 18:53:15: Cluster-wide one-time actions... Done!
    2013-07-10 18:53:15: Oracle CRS home = /u01/grid_home
    2013-07-10 18:53:15: Host name = rac2
    2013-07-10 18:53:15: CRS user = grid
    2013-07-10 18:53:15: Oracle CRS home = /u01/grid_home
    2013-07-10 18:53:15: GPnP host = rac2
    2013-07-10 18:53:15: Oracle GPnP home = /u01/grid_home/gpnp
    2013-07-10 18:53:15: Oracle GPnP local home = /u01/grid_home/gpnp/rac2
    2013-07-10 18:53:15: GPnP directories verified.
    2013-07-10 18:53:15: Checking to see if Oracle CRS stack is already configured
    2013-07-10 18:53:15: Oracle CRS stack is not configured yet
    2013-07-10 18:53:15: ---Checking local gpnp setup...
    2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml" does not exist
    2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso" does not exist
    2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso" does not exist
    2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp/rac2: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
    2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp/rac2: INVALID (bad profile/wallet)
    2013-07-10 18:53:15: ---Checking cluster-wide gpnp setup...
    2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1
    2013-07-10 18:53:15: gpnptool: run /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer
    2013-07-10 18:53:15: Running as user grid: /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer
    2013-07-10 18:53:15: s_run_as_user2: Running /bin/su grid -c ' /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer '
    2013-07-10 18:53:15: Removing file /tmp/file0qKE0c
    2013-07-10 18:53:15: Successfully removed file: /tmp/file0qKE0c
    2013-07-10 18:53:15: /bin/su successfully executed
    2013-07-10 18:53:15: gpnptool: rc=0
    2013-07-10 18:53:15: gpnptool output:
    Profile signature is valid.
    2013-07-10 18:53:15: Profile "/u01/grid_home/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/grid_home/gpnp/wallets/peer"
    2013-07-10 18:53:15: gpnptool: run /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer
    2013-07-10 18:53:15: Running as user grid: /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer
    2013-07-10 18:53:15: s_run_as_user2: Running /bin/su grid -c ' /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer '
    2013-07-10 18:53:16: Removing file /tmp/filebkOtBv
    2013-07-10 18:53:16: Successfully removed file: /tmp/filebkOtBv
    2013-07-10 18:53:16: /bin/su successfully executed
    2013-07-10 18:53:16: gpnptool: rc=0
    2013-07-10 18:53:16: gpnptool output:
    Profile signature is valid.
    2013-07-10 18:53:16: Profile "/u01/grid_home/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/grid_home/gpnp/wallets/prdr"
    2013-07-10 18:53:16: chk gpnphome /u01/grid_home/gpnp: OK
    2013-07-10 18:53:16: GPnP Wallets ownership/permissions successfully set.
    2013-07-10 18:53:16: gpnp setup checked: local valid? 0 cluster-wide valid? 1
    2013-07-10 18:53:16: Taking cluster-wide setup as local
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/profiles/peer/profile.xml" => "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/peer/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/prdr/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/profiles/peer/profile_orig.xml" => "/u01/grid_home/gpnp/rac2/profiles/peer/profile_orig.xml"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/profiles/peer/profile_orig.xml" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/root/ewallet.p12" => "/u01/grid_home/gpnp/rac2/wallets/root/ewallet.p12"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/root/ewallet.p12" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/pa/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/pa/cwallet.sso"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/pa/cwallet.sso" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/root/b64certificate.txt" => "/u01/grid_home/gpnp/rac2/wallets/root/b64certificate.txt"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/root/b64certificate.txt" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/peer/cert.txt" => "/u01/grid_home/gpnp/rac2/wallets/peer/cert.txt"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/peer/cert.txt" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/pa/cert.txt" => "/u01/grid_home/gpnp/rac2/wallets/pa/cert.txt"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/pa/cert.txt" => (grid,oinstall)
    2013-07-10 18:53:16: GPnP Wallets ownership/permissions successfully set.
    2013-07-10 18:53:16: gpnp setup: GOTCLUSTERWIDE
    2013-07-10 18:53:16: Validating for SI-CSS configuration
    2013-07-10 18:53:16: Retrieving OCR main disk location
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc
    2013-07-10 18:53:16: Unable to retrieve ocr disk info
    2013-07-10 18:53:16: Checking to see if any 9i GSD is up
    2013-07-10 18:53:16: libskgxnBase_lib = /etc/ORCLcluster/oracm/lib/libskgxn2.so
    2013-07-10 18:53:16: libskgxn_lib = /opt/ORCLcluster/lib/libskgxn2.so
    2013-07-10 18:53:16: SKGXN library file does not exists
    2013-07-10 18:53:16: OLR location = /u01/grid_home/cdata/rac2.olr
    2013-07-10 18:53:16: Oracle CRS Home = /u01/grid_home
    2013-07-10 18:53:16: Validating /etc/oracle/olr.loc file for OLR location /u01/grid_home/cdata/rac2.olr
    2013-07-10 18:53:16: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig
    2013-07-10 18:53:16: Oracle CRS home = /u01/grid_home
    2013-07-10 18:53:16: Oracle cluster name = rac-scan
    2013-07-10 18:53:16: OCR locations = +CRS
    2013-07-10 18:53:16: Validating OCR
    2013-07-10 18:53:16: Retrieving OCR location used by previous installations
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrmirrorconfig_loc
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc3
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc4
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc5
    2013-07-10 18:53:16: Checking if OCR sync file exists
    2013-07-10 18:53:16: No need to sync OCR file
    2013-07-10 18:53:16: OCR_LOCATION=+CRS
    2013-07-10 18:53:16: OCR_MIRROR_LOCATION=
    2013-07-10 18:53:16: OCR_MIRROR_LOC3=
    2013-07-10 18:53:16: OCR_MIRROR_LOC4=
    2013-07-10 18:53:16: OCR_MIRROR_LOC5=
    2013-07-10 18:53:16: Current OCR location=
    2013-07-10 18:53:16: Current OCR mirror location=
    2013-07-10 18:53:16: Current OCR mirror loc3=
    2013-07-10 18:53:16: Current OCR mirror loc4=
    2013-07-10 18:53:16: Current OCR mirror loc5=
    2013-07-10 18:53:16: Verifying current OCR settings with user entered values
    2013-07-10 18:53:16: Setting OCR locations in /etc/oracle/ocr.loc
    2013-07-10 18:53:16: Validating OCR locations in /etc/oracle/ocr.loc
    2013-07-10 18:53:16: Checking for existence of /etc/oracle/ocr.loc
    2013-07-10 18:53:16: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig
    2013-07-10 18:53:16: Setting ocr location +CRS
    2013-07-10 18:53:16: Creating or upgrading Oracle Local Registry (OLR)
    2013-07-10 18:53:16: OLR successfully created or upgraded
    2013-07-10 18:53:16: /u01/grid_home/bin/clscfg -localadd
    2013-07-10 18:53:16: Keys created in the OLR successfully
    2013-07-10 18:53:16: GPnP setup state: new-cluster-wide
    2013-07-10 18:53:16: GPnP cluster configuration already performed
    2013-07-10 18:53:16: Registering ohasd
    2013-07-10 18:53:16: init file = /u01/grid_home/crs/init/init.ohasd
    2013-07-10 18:53:16: Copying file /u01/grid_home/crs/init/init.ohasd to /etc/init.d directory
    2013-07-10 18:53:16: Setting init.ohasd permission in /etc/init.d directory
    2013-07-10 18:53:16: init file = /u01/grid_home/crs/init/ohasd
    2013-07-10 18:53:16: Copying file /u01/grid_home/crs/init/ohasd to /etc/init.d directory
    2013-07-10 18:53:16: Setting ohasd permission in /etc/init.d directory
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc3.d/S96ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc3.d/S96ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc5.d/S96ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc5.d/S96ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc0.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc0.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc1.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc1.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc2.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc2.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc4.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc4.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc6.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc6.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: The file ohasd has been successfully linked to the RC directories
    2013-07-10 18:53:16: Starting ohasd
    2013-07-10 18:53:16: itab entries=
    2013-07-10 18:53:21: Created backup /etc/inittab.no_crs
    2013-07-10 18:53:21: Appending to /etc/inittab.tmp:
    2013-07-10 18:53:21: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
    2013-07-10 18:53:21: Done updating /etc/inittab.tmp
    2013-07-10 18:53:21: Saved /etc/inittab.crs
    2013-07-10 18:53:21: Installed new /etc/inittab
    2013-07-10 18:53:36: ohasd is starting
    2013-07-10 18:53:36: Checking ohasd
    2013-07-10 18:53:37: ohasd started successfully
    2013-07-10 18:53:37: Creating CRS resources and dependencies
    2013-07-10 18:53:37: Configuring HASD
    2013-07-10 18:53:37: Registering type ora.daemon.type
    2013-07-10 18:53:37: Registering type ora.mdns.type
    2013-07-10 18:53:37: Registering type ora.gpnp.type
    2013-07-10 18:53:38: Registering type ora.gipc.type
    2013-07-10 18:53:38: Registering type ora.cssd.type
    2013-07-10 18:53:38: Registering type ora.cssdmonitor.type
    2013-07-10 18:53:39: Registering type ora.crs.type
    2013-07-10 18:53:39: Registering type ora.evm.type
    2013-07-10 18:53:39: Registering type ora.ctss.type
    2013-07-10 18:53:40: Registering type ora.asm.type
    2013-07-10 18:53:40: Registering type ora.drivers.acfs.type
    2013-07-10 18:53:40: Registering type ora.diskmon.type
    2013-07-10 18:53:51: ADVM/ACFS is configured
    2013-07-10 18:53:51: Successfully created CRS resources for cluster daemon and ASM
    2013-07-10 18:53:51: Checking if initial configuration has been performed
    2013-07-10 18:53:51: Starting CSS in exclusive mode
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    2013-07-10 18:54:19: Querying for existing CSS voting disks
    2013-07-10 18:54:19: Performing initial configuration for cluster
    2013-07-10 18:54:21: Start of resource "ora.ctssd -init" Succeeded
    2013-07-10 18:54:21: Configuring ASM via ASMCA
    2013-07-10 18:54:21: Executing as grid: /u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM
    2013-07-10 18:54:21: Running as user grid: /u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM
    2013-07-10 18:54:21:   Invoking "/u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM" as user "grid"
    2013-07-10 18:54:40: Configuration of ASM failed, see logs for details
    2013-07-10 18:54:40: Did not succssfully configure and start ASM
    2013-07-10 18:54:40: Exiting exclusive mode
    2013-07-10 18:54:40: Command return code of 1 (256) from command: /u01/grid_home/bin/crsctl stop resource ora.crsd -init
    2013-07-10 18:54:40: Stop of resource "ora.crsd -init" failed
    2013-07-10 18:54:40: Failed to stop CRSD
    2013-07-10 18:55:04: Initial cluster configuration failed.  See /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log for details
    Also below are some of the configs related to rac2 node
    [root@rac2 rac2]# rpm -qa | grep oracleasm
    oracleasmlib-2.0.4-1.el5
    oracleasm-support-2.1.8-1.el5
    oracleasm-2.6.18-274.el5xen-2.0.5-1.el5
    oracleasm-2.6.18-274.el5-2.0.5-1.el5
    oracleasm-2.6.18-274.el5debug-2.0.5-1.el5
    oracleasm-2.6.18-274.el5-debuginfo-2.0.5-1.el5
    [root@rac2 rac2]# /usr/sbin/oracleasm configure
    ORACLEASM_ENABLED=true
    ORACLEASM_UID=grid
    ORACLEASM_GID=asmadmin
    ORACLEASM_SCANBOOT=true
    ORACLEASM_SCANORDER=""
    ORACLEASM_SCANEXCLUDE=""
    ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"
    [root@rac2 rac2]# /usr/sbin/oracleasm status
    Checking if ASM is loaded: yes
    Checking if /dev/oracleasm is mounted: yes
    [root@rac2 rac2]# /usr/sbin/oracleasm listdisks
    CRS1
    DATA1
    FRA1
    [root@rac2 rac2]# ls -l /dev/oracleasm/disks/
    total 0
    brw-rw---- 1 grid asmadmin 8, 17 Jul 10 18:35 CRS1
    brw-rw---- 1 grid asmadmin 8, 33 Jul 10 18:36 DATA1
    brw-rw---- 1 grid asmadmin 8, 49 Jul 10 18:36 FRA1
    [root@rac2 rac2]# cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1               localhost.localdomain localhost
    ::1             localhost6.localdomain6 localhost6
    #Public IP's(eth0)
    192.168.0.101    rac1.naveed.com    rac1
    192.168.0.102    rac2.naveed.com    rac2
    #Private IP's(eth1)
    192.168.1.101    rac1-prv.naveed.com   rac1-prv
    192.168.1.102    rac2-prv.naveed.com   rac2-prv
    #VIPS
    192.168.0.221    rac1-vip.naveed.com   rac1-vip
    192.168.0.222    rac2-vip.naveed.com   rac2-vip
    #DNS server IP
    192.168.0.10    naveeddns.naveed.com   naveeddns
    [root@rac2 rac2]#
    Thanks in advance

    Hi,
    First of all thanks a lot for the response. You wont't beleive this is my 7th fresh installation and everytime in node 2 i m hit with this same error.
    Also i tried below procedure instead of fresh installation
    once i deconfig & rerun (./rootcrs.pl -verbose -deconfig -force) on node 2
    Using configuration parameter file: ./crsconfig_params
    PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
    PRCR-1068 : Failed to query resources
    Cannot communicate with crsd
    PRCR-1070 : Failed to check if resource ora.gsd is registered
    Cannot communicate with crsd
    PRCR-1070 : Failed to check if resource ora.ons is registered
    Cannot communicate with crsd
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4000: Command Stop failed, or completed with errors.
    CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
    CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac2'
    CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
    CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
    CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'
    CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
    CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
    CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
    CRS-2677: Stop of 'ora.drivers.acfs' on 'rac2' succeeded
    CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
    CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
    CRS-4133: Oracle High Availability Services has been stopped.
    Successfully deconfigured Oracle clusterware stack on this node
    [root@rac2 grid_home]# ./root.sh
    Performing root user operation for Oracle 11g
    The following environment variables are set as:
        ORACLE_OWNER= grid
        ORACLE_HOME=  /u01/grid_home
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
    The contents of "dbhome" have not changed. No need to overwrite.
    The contents of "oraenv" have not changed. No need to overwrite.
    The contents of "coraenv" have not changed. No need to overwrite.
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root script.
    Now product-specific root actions will be performed.
    Using configuration parameter file: /u01/grid_home/crs/install/crsconfig_params
    User ignored Prerequisites during installation
    OLR initialization - successful
    Adding Clusterware entries to inittab
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    ASM created and started successfully.
    Disk Group CRS mounted successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Successful addition of voting disk 636af26485ef4f27bfec31523aaa0660.
    Successfully replaced voting disk group with +CRS.
    CRS-4266: Voting file(s) successfully replaced
    ##  STATE    File Universal Id                File Name Disk group
    1. ONLINE   636af26485ef4f27bfec31523aaa0660 (ORCL:CRS1) [CRS]
    Located 1 voting disk(s).
    Start of resource "ora.crsd" failed
    CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'rac2'
    CRS-4000: Command Start failed, or completed with errors.
    Failed to start Oracle Grid Infrastructure stack
    Failed to start Cluster Ready Services at /u01/grid_home/crs/install/crsconfig_lib.pm line 1286.
    /u01/grid_home/perl/bin/perl -I/u01/grid_home/perl/lib -I/u01/grid_home/crs/install /u01/grid_home/crs/install/rootcrs.pl execution failed

  • 11gR2 clusterware installation problem on root.sh script on second node

    Hi all,
    I wanna install the *11gR2 RAC* on ORA-Linux 5.5 (x86_64) using VMware server but on the second node i get two "*failed*" at the end of root.sh script.
    After that i try to install DB but ı can see only one node.What is the problem...
    I will send the output, ı need your help.
    Thank you all for helping..
    Hosts file:(we have no ping problem )
    [root@rac2 ~]# cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1       localhost.localdomain   localhost
    # Public
    192.168.2.101   rac1.localdomain        rac1
    192.168.2.102   rac2.localdomain        rac2
    # Private
    192.168.0.101   rac1-priv.localdomain   rac1-priv
    192.168.0.102   rac2-priv.localdomain   rac2-priv
    # Virtual
    192.168.2.111   rac1-vip.localdomain    rac1-vip
    192.168.2.112   rac2-vip.localdomain    rac2-vip
    # SCAN
    192.168.2.201   rac-scan.localdomain rac-scan
    [root@rac2 ~]#
    FIRST NODE root.sh script output...
    [root@rac2 ~]# /u01/app/11.2.0/db_1/root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /u01/app/11.2.0/db_1
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
       Copying dbhome to /usr/local/bin ...
       Copying oraenv to /usr/local/bin ...
       Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-12-06 14:45:06: Parsing the host name
    2010-12-06 14:45:06: Checking for super user privileges
    2010-12-06 14:45:06: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/db_1/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
      root wallet
      root wallet cert
      root cert export
      peer wallet
      profile reader wallet
      pa wallet
      peer wallet keys
      pa wallet keys
      peer cert request
      pa cert request
      peer cert
      pa cert
      peer root cert TP
      profile reader root cert TP
      pa root cert TP
      peer pa cert TP
      pa peer cert TP
      profile reader pa cert TP
      profile reader peer cert TP
      peer user cert
      pa user cert
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
    CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
    ASM created and started successfully.
    DiskGroup DATA created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
    CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
    CRS-4256: Updating the profile
    Successful addition of voting disk 587cc69413ce4fd3bf0c2c2548fb9017.
    Successfully replaced voting disk group with +DATA.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ##  STATE    File Universal Id                File Name Disk group
    1. ONLINE   587cc69413ce4fd3bf0c2c2548fb9017 (/dev/oracleasm/disks/DISK1) [DATA]
    Located 1 voting disk(s).
    CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
    CRS-2677: Stop of 'ora.crsd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
    CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
    CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac2'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
    CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
    CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
    CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
    CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
    CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'rac2'
    CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
    CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 'rac2'
    CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'rac2'
    CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac2'
    CRS-2676: Start of 'ora.DATA.dg' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac2'
    CRS-2676: Start of 'ora.registry.acfs' on 'rac2' succeeded
    rac2     2010/12/06 14:52:06     /u01/app/11.2.0/db_1/cdata/rac2/backup_20101206_145206.olr
    Configure Oracle Grid Infrastructure for a Cluster ... succeeded
    Updating inventory properties for clusterware
    Starting Oracle Universal Installer...
    Checking swap space: must be greater than 500 MB.   Actual 6847 MB    Passed
    The inventory pointer is located at /etc/oraInst.loc
    The inventory is located at /u01/app/oraInventory
    'UpdateNodeList' was successful.
    [root@rac2 ~]#
    SECOND NODE root.sh script output
    [root@rac1 db_1]# ./root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /u01/app/11.2.0/db_1
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
       Copying dbhome to /usr/local/bin ...
       Copying oraenv to /usr/local/bin ...
       Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-12-06 14:54:11: Parsing the host name
    2010-12-06 14:54:11: Checking for super user privileges
    2010-12-06 14:54:11: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/db_1/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
    CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
    CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
    CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
    CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
    CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
    ASM created and started successfully.
    DiskGroup DATA created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
    CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
    Successful addition of voting disk 2761ce8d47b44fbabf73462151e3ba1d.
    Successfully replaced voting disk group with +DATA.
    CRS-4266: Voting file(s) successfully replaced
    ##  STATE    File Universal Id                File Name Disk group
    1. ONLINE   2761ce8d47b44fbabf73462151e3ba1d (/dev/oracleasm/disks/DISK1) [DATA]
    Located 1 voting disk(s).
    CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
    CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
    CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
    CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
    CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
    CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
    CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
    CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
    CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
    CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
    CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
    CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
    CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'rac1'
    CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
    CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
    CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'rac1'
    CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
    CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'
    CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded
    PRCR-1079 : *Failed* to start resource ora.scan1.vip
    CRS-5005: IP Address: 192.168.2.201 is already in use in the network
    CRS-2674: Start of 'ora.scan1.vip' on 'rac1' *failed*
    CRS-2632: There are no more servers to try to place resource 'ora.scan1.vip' on that would satisfy its placement policy
    start scan ... *failed*
    Configure Oracle Grid Infrastructure for a Cluster ... *failed*
    Updating inventory properties for clusterware
    Starting Oracle Universal Installer...
    Checking swap space: must be greater than 500 MB.   Actual 6847 MB    Passed
    The inventory pointer is located at /etc/oraInst.loc
    The inventory is located at /u01/app/oraInventory
    'UpdateNodeList' was successful.
    [root@rac1 db_1]# * "./runcluvfy.sh stage -pre -crsinst -n rac1,rac2 " outputs are same each node....*
    [oracle@rac2 grid]$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2
    Performing pre-checks for cluster services setup
    Checking node reachability...
    Node reachability check passed from node "rac2"
    Checking user equivalence...
    User equivalence check passed for user "oracle"
    Checking node connectivity...
    Checking hosts config file...
    Verification of the hosts config file successful
    Node connectivity passed for subnet "192.168.2.0" with node(s) rac2,rac1
    TCP connectivity check passed for subnet "192.168.2.0"
    Node connectivity passed for subnet "192.168.122.0" with node(s) rac2,rac1
    TCP connectivity check failed for subnet "192.168.122.0"
    Node connectivity passed for subnet "192.168.0.0" with node(s) rac2,rac1
    TCP connectivity check passed for subnet "192.168.0.0"
    Interfaces found on subnet "192.168.2.0" that are likely candidates for VIP are:
    rac2 eth0:192.168.2.102 eth0:192.168.2.112 eth0:192.168.2.201
    rac1 eth0:192.168.2.101 eth0:192.168.2.111
    Interfaces found on subnet "192.168.122.0" that are likely candidates for a private interconnect are:
    rac2 virbr0:192.168.122.1
    rac1 virbr0:192.168.122.1
    Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect are:
    rac2 eth1:192.168.0.102
    rac1 eth1:192.168.0.101
    Node connectivity check passed
    Total memory check passed
    Available memory check passed
    Swap space check passed
    Free disk space check passed for "rac2:/tmp"
    Free disk space check passed for "rac1:/tmp"
    User existence check passed for "oracle"
    Group existence check passed for "oinstall"
    Group existence check passed for "dba"
    Membership check for user "oracle" in group "oinstall" [as Primary] passed
    Membership check for user "oracle" in group "dba" passed
    Run level check passed
    Hard limits check passed for "maximum open file descriptors"
    Soft limits check passed for "maximum open file descriptors"
    Hard limits check passed for "maximum user processes"
    Soft limits check passed for "maximum user processes"
    System architecture check passed
    Kernel version check passed
    Kernel parameter check passed for "semmsl"
    Kernel parameter check passed for "semmns"
    Kernel parameter check passed for "semopm"
    Kernel parameter check passed for "semmni"
    Kernel parameter check passed for "shmmax"
    Kernel parameter check passed for "shmmni"
    Kernel parameter check passed for "shmall"
    Kernel parameter check passed for "file-max"
    Kernel parameter check passed for "ip_local_port_range"
    Kernel parameter check passed for "rmem_default"
    Kernel parameter check passed for "rmem_max"
    Kernel parameter check passed for "wmem_default"
    Kernel parameter check passed for "wmem_max"
    Kernel parameter check passed for "aio-max-nr"
    Package existence check passed for "make-3.81"
    Package existence check passed for "binutils-2.17.50.0.6"
    Package existence check passed for "gcc-4.1.2"
    Package existence check passed for "libaio-0.3.106 (i386)"
    Package existence check passed for "libaio-0.3.106 (x86_64)"
    Package existence check passed for "glibc-2.5-24 (i686)"
    Package existence check passed for "glibc-2.5-24 (x86_64)"
    Package existence check passed for "compat-libstdc++-33-3.2.3 (i386)"
    Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)"
    Package existence check passed for "elfutils-libelf-0.125 (x86_64)"
    Package existence check passed for "elfutils-libelf-devel-0.125"
    Package existence check passed for "glibc-common-2.5"
    Package existence check passed for "glibc-devel-2.5 (i386)"
    Package existence check passed for "glibc-devel-2.5 (x86_64)"
    Package existence check passed for "glibc-headers-2.5"
    Package existence check passed for "gcc-c++-4.1.2"
    Package existence check passed for "libaio-devel-0.3.106 (i386)"
    Package existence check passed for "libaio-devel-0.3.106 (x86_64)"
    Package existence check passed for "libgcc-4.1.2 (i386)"
    Package existence check passed for "libgcc-4.1.2 (x86_64)"
    Package existence check passed for "libstdc++-4.1.2 (i386)"
    Package existence check passed for "libstdc++-4.1.2 (x86_64)"
    Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)"
    Package existence check passed for "sysstat-7.0.2"
    Package existence check passed for "unixODBC-2.2.11 (i386)"
    Package existence check passed for "unixODBC-2.2.11 (x86_64)"
    Package existence check passed for "unixODBC-devel-2.2.11 (i386)"
    Package existence check passed for "unixODBC-devel-2.2.11 (x86_64)"
    Package existence check passed for "ksh-20060214"
    Check for multiple users with UID value 0 passed
    Current group ID check passed
    Core file name pattern consistency check passed.
    User "oracle" is not part of "root" group. Check passed
    Default user file creation mask check passed
    Starting Clock synchronization checks using Network Time Protocol(NTP)...
    NTP Configuration file check started...
    NTP Configuration file check passed
    Checking daemon liveness...
    Liveness check passed for "ntpd"
    NTP daemon slewing option check passed
    NTP daemon's boot time configuration check for slewing option passed
    NTP common Time Server Check started...
    Check of common NTP Time Server passed
    Clock time offset check from NTP Time Server started...
    Clock time offset check passed
    Clock synchronization check using Network Time Protocol(NTP) passed
    Pre-check for cluster services setup was successful.
    [oracle@rac2 grid]$ I'm confused :)
    Edited by: Eren GULERYUZ on 06.Ara.2010 05:57

    Hi,
    it looks like, that your "shared device" you are using is not really shared.
    The second node does "create an ASM diskgroup" and create OCR and Voting disks. If this indeed would be a shared device, he should have recognized, that your disk is shared.
    So as a result your VMware configuration must be wrong, and the disk you presented as shared disk is not really shared.
    Which VMWare version did you use? It will not work correctly with the workstation or player edition, since shared disks are only really working with the server version.
    If you indeed using the server, could you paste your vm configurations?
    Furthermore I recommend using Virtual Box. There is a nice how-to:
    http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnOEL5UsingVirtualBox.php
    Sebastian

  • RAC, ASM failed to start up on second node , ORA-03113: end-of-file on comm

    i'm installing an RAC with 2 nodes on top of ASM
    when creating ASM Diskgroup , it failed and reported error CRS-0215 failed to start asm on node2
    Oracle 10.2.0.1
    linux CentOs 4.x
    u01/app/oracle/product/10.2.0/db_1/bin/dbca  -progress_only   -configureASM -templateName NO_VALUE -gdbName NO -sid NO      -emConf
    iguration NONE    -diskList /dev/raw/raw2,/dev/raw/raw3  -diskGroupName DATA -datafileJarLocation /u01/app/oracle/product/10.2.0/db_
    1/assistants/dbca/templates  -responseFile NO_VALUE  -nodeinfo node1,node2    -obfuscatedPasswords true   -oratabLocation /u01/app/o
    racle/product/10.2.0/db_1/install/oratab   -asmSysPassword 05dbb0be38ecf8cca822cf3cf99e675448  -redundancy EXTERNA
    [oracle@node2 bin]$ ./crs_stat -t -v
    Name           Type           R/RA   F/FT   Target    State     Host       
    ora....SM1.asm application    0/5    0/0    ONLINE    ONLINE    node1      
    ora....E1.lsnr application    0/5    0/0    ONLINE    ONLINE    node1      
    ora.node1.gsd  application    0/5    0/0    ONLINE    ONLINE    node1      
    ora.node1.ons  application    0/3    0/0    ONLINE    ONLINE    node1      
    ora.node1.vip  application    0/0    0/0    ONLINE    ONLINE    node1      
    ora....SM2.asm application    0/5    0/0    OFFLINE   OFFLINE              
    ora....E2.lsnr application    0/5    0/0    ONLINE    ONLINE    node2      
    ora.node2.gsd  application    0/5    0/0    ONLINE    ONLINE    node2      
    ora.node2.ons  application    0/3    0/0    ONLINE    ONLINE    node2      
    ora.node2.vip  application    0/0    0/0    ONLINE    ONLINE    node2  
    i checked the status , asm is able to start on both nodes if not at the same time ,
    when trying to start the second node , with srvctl or sqlplus , each give the error 03113
    can anyone suggest me of how to bring up both instances ,
    thanks~
    [oracle@node2 bin]$ srvctl stop asm -n node1
    [oracle@node2 bin]$ srvctl start asm -n node1
    [oracle@node2 bin]$ srvctl start asm -n node2
    PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Wed May 27 16:14:50 2009
    node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:Copyright (c) 1982, 2005, Oracle.  All rights reserved.
    node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:Enter user-name: Connected to an idle instance.
    node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:SQL> ORA-03113: end-of-file on communication channel
    node2:ora.node2.ASM2.asm:SQL> Disconnected
    node2:ora.node2.ASM2.asm:
    [code/]
    Edited by: zs_hzh on May 27, 2009 1:25 AM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

    Is it possible to start ASM on second node with SQL*Plus in NOMOUNT state?

  • Root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource

    Dear all,
    I am trying to install clean Oracle 11.2.0.3 grid infrastructure on a two node cluster running on Solaris 5.10.
    - Cluster verification was successfully on both nodes; No warning or issues;
    - I am using 2 network cards for the public and 2 for the private interconnect;
    - OCR is stored on ASM
    - Firewall is disabled on both nodes
    - SCAN is being configured on the DNS (not added in /etc/hosts)
    - GNS is not used
    - hosts file is identical (except the primary hostname)
    The problem: root.sh fails on the 2nd (remote) node, because it fails to start the "ora.asm" resource. However, the root.sh has completed successfully on the 1st node.. Somehow, root.sh doesn't create +ASM2 instance on the remote (host2) node.
    root.sh was executed first on the local node (host1) and after the successful execution was started on the remote (host2) node.
    Output from host1 (working):
    ===================
    Adding Clusterware entries to inittab
    CRS-2672: Attempting to start 'ora.mdnsd' on 'host1'
    CRS-2676: Start of 'ora.mdnsd' on 'host1' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'host1'
    CRS-2676: Start of 'ora.gpnpd' on 'host1' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'host1'
    CRS-2672: Attempting to start 'ora.gipcd' on 'host1'
    CRS-2676: Start of 'ora.cssdmonitor' on 'host1' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'host1' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'host1'
    CRS-2672: Attempting to start 'ora.diskmon' on 'host1'
    CRS-2676: Start of 'ora.diskmon' on 'host1' succeeded
    CRS-2676: Start of 'ora.cssd' on 'host1' succeeded
    ASM created and started successfully.
    Disk Group CRS created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    CRS-4256: Updating the profile
    Successful addition of voting disk 4373be34efab4f01bf79f6c5362acfd3.
    Successful addition of voting disk 7fd725fa4d904f07bf76cecf96791547.
    Successful addition of voting disk a9c85297bdd74f3abfd86899205aaf17.
    Successfully replaced voting disk group with +CRS.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ## STATE File Universal Id File Name Disk group
    1. ONLINE 4373be34efab4f01bf79f6c5362acfd3 (/dev/rdsk/c4t600A0B80006E2CC40000C6674E82AA57d0s4) [CRS]
    2. ONLINE 7fd725fa4d904f07bf76cecf96791547 (/dev/rdsk/c4t600A0B80006E2CC40000C6694E82AADDd0s4) [CRS]
    3. ONLINE a9c85297bdd74f3abfd86899205aaf17 (/dev/rdsk/c4t600A0B80006E2F100000C7744E82AC7Ad0s4) [CRS]
    Located 3 voting disk(s).
    CRS-2672: Attempting to start 'ora.asm' on 'host1'
    CRS-2676: Start of 'ora.asm' on 'host1' succeeded
    CRS-2672: Attempting to start 'ora.CRS.dg' on 'host1'
    CRS-2676: Start of 'ora.CRS.dg' on 'host1' succeeded
    CRS-2672: Attempting to start 'ora.registry.acfs' on 'host1'
    CRS-2676: Start of 'ora.registry.acfs' on 'host1' succeeded
    Configure Oracle Grid Infrastructure for a Cluster ... succeeded
    Name Type Target State Host
    ora.CRS.dg ora....up.type ONLINE ONLINE host1
    ora....ER.lsnr ora....er.type ONLINE ONLINE host1
    ora....N1.lsnr ora....er.type ONLINE ONLINE host1
    ora....N2.lsnr ora....er.type ONLINE ONLINE host1
    ora....N3.lsnr ora....er.type ONLINE ONLINE host1
    ora.asm ora.asm.type ONLINE ONLINE host1
    ora....SM1.asm application ONLINE ONLINE host1
    ora....B1.lsnr application ONLINE ONLINE host1
    ora....db1.gsd application OFFLINE OFFLINE
    ora....db1.ons application ONLINE ONLINE host1
    ora....db1.vip ora....t1.type ONLINE ONLINE host1
    ora.cvu ora.cvu.type ONLINE ONLINE host1
    ora.gsd ora.gsd.type OFFLINE OFFLINE
    ora....network ora....rk.type ONLINE ONLINE host1
    ora.oc4j ora.oc4j.type ONLINE ONLINE host1
    ora.ons ora.ons.type ONLINE ONLINE host1
    ora....ry.acfs ora....fs.type ONLINE ONLINE host1
    ora.scan1.vip ora....ip.type ONLINE ONLINE host1
    ora.scan2.vip ora....ip.type ONLINE ONLINE host1
    ora.scan3.vip ora....ip.type ONLINE ONLINE host1
    Output from host2 (failing):
    ===================
    OLR initialization - successful
    Adding Clusterware entries to inittab
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node billdb1, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    Start of resource "ora.asm" failed
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'host2'
    CRS-2676: Start of 'ora.drivers.acfs' on 'host2' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'host2'
    CRS-5017: The resource action "ora.asm start" encountered the following error:
    ORA-03113: end-of-file on communication channel
    Process ID: 0
    Session ID: 0 Serial number: 0
    *. For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log".*
    CRS-2674: Start of 'ora.asm' on 'host2' failed
    CRS-2679: Attempting to clean 'ora.asm' on 'host2'
    CRS-2681: Clean of 'ora.asm' on 'host2' succeeded
    CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'host2'
    CRS-2677: Stop of 'ora.drivers.acfs' on 'host2' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    Failed to start Oracle Grid Infrastructure stack
    Failed to start ASM at /u01/11.2.0/grid/crs/install/crsconfig_lib.pm line 1272.
    /u01/11.2.0/grid/perl/bin/perl -I/u01/11.2.0/grid/perl/lib -I/u01/11.2.0/grid/crs/install /u01/11.2.0/grid/crs/install/rootcrs.pl execution failed
    Contents of "/u01/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_host2.log"
    =============================================
    CRS-2672: Attempting to start 'ora.asm' on 'host2'
    CRS-5017: The resource action "ora.asm start" encountered the following error:
    ORA-03113: end-of-file on communication channel
    Process ID: 0
    Session ID: 0 Serial number: 0
    . For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log".
    CRS-2674: Start of 'ora.asm' on 'host2' failed
    CRS-2679: Attempting to clean 'ora.asm' on 'host2'
    CRS-2681: Clean of 'ora.asm' on 'host2' succeeded
    CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'host2'
    CRS-2677: Stop of 'ora.drivers.acfs' on 'host2' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    2011-10-24 19:36:54: Failed to start Oracle Grid Infrastructure stack
    2011-10-24 19:36:54: ###### Begin DIE Stack Trace ######
    2011-10-24 19:36:54: Package File Line Calling
    2011-10-24 19:36:54: --------------- -------------------- ---- ----------
    2011-10-24 19:36:54: 1: main rootcrs.pl 375 crsconfig_lib::dietrap
    2011-10-24 19:36:54: 2: crsconfig_lib crsconfig_lib.pm 1272 main::__ANON__
    2011-10-24 19:36:54: 3: crsconfig_lib crsconfig_lib.pm 1171 crsconfig_lib::start_cluster
    2011-10-24 19:36:54: 4: main rootcrs.pl 803 crsconfig_lib::perform_start_cluster
    2011-10-24 19:36:54: ####### End DIE Stack Trace #######
    Shortened output from "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] clean {
    2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] InstAgent::stop_option stop mode immediate option 1
    2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] InstAgent::stop {
    2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] InstAgent::stop original reason system do shutdown abort
    2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] ConnectionPool::resetConnection s_statusOfConnectionMap 00ab1948
    2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] ConnectionPool::resetConnection sid +ASM2 status  2
    2011-10-24 19:35:48.728: [ora.asm][9] {0:0:224} [start] Gimh::check OH /u01/11.2.0/grid SID +ASM2
    2011-10-24 19:35:48.728: [ora.asm][9] {0:0:224} [start] Gimh::check condition changes to (GIMH_NEXT_NUM) 0,1,7 exists
    2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] (:CLSN00006:)AsmAgent::check failed gimh state 0
    2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 0
    2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet { entry
    2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet procr_get_conf: retval [0] configured [1] local only [0] error buffer []
    2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet procr_get_conf: OCR loc [0], Disk Group : [+CRS]
    2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet m_ocrDgpSet 015fba90 dgName CRS
    2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet ocrret 0 found 1
    2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet ocrDgpSet CRS
    2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet exit }
    2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::ocrDgCheck Entry {
    2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] DgpAgent::getConnxn new pool
    2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] DgpAgent::getConnxn new pool m_oracleHome:/u01/11.2.0/grid m_oracleSid:+ASM2 m_usrOraEnv:
    2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] ConnectionPool::ConnectionPool 2 m_oracleHome:/u01/11.2.0/grid, m_oracleSid:+ASM2, m_usrOraEnv:
    2011-10-24 19:35:48.733: [ora.asm][9] {0:0:224} [start] ConnectionPool::addConnection m_oracleHome:/u01/11.2.0/grid m_oracleSid:+ASM2 m_usrOraEnv: pConnxn:
    01fcdf10
    2011-10-24 19:35:48.733: [ora.asm][9] {0:0:224} [start] Utils::getCrsHome crsHome /u01/11.2.0/grid
    2011-10-24 19:35:51.969: [ora.asm][14] {0:0:224} [check] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/11.2.0/grid/bin/oracle)(ARGV0=o
    racle+ASM2)(ENVS='ORACLE_HOME=/u01/11.2.0/grid,ORACLE_SID=+ASM2')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM2)))
    2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] ConnectionPool::getConnection 260 pConnxn 013e40a0
    2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] DgpAgent::getConnxn connected
    2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] InstConnection::connectInt: server not attached
    2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    SVR4 Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] InstConnection::connectInt (2) Exception OCIException
    2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] InstConnection:connect:excp OCIException OCI error 1034
    2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus excp ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    SVR4 Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus asm inst is down or going down
    2011-10-24 19:35:52.191: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus dgName CRS ret 1
    2011-10-24 19:35:52.191: [ora.asm][14] {0:0:224} [check] (:CLSN00100:)DgpAgent::ocrDgCheck OCR dgName CRS state 1
    2011-10-24 19:35:52.192: [ora.asm][14] {0:0:224} [check] ConnectionPool::releaseConnection InstConnection 013e40a0
    2011-10-24 19:35:52.192: [ora.asm][14] {0:0:224} [check] AsmAgent::check ocrCheck 2 m_OcrOnline 0 m_OcrTimer 0
    2011-10-24 19:35:52.193: [ora.asm][14] {0:0:224} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
    2011-10-24 19:35:52.197: [ COMMCRS][23]clsc_connect: (1020d39d0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))
    Please advice for any workaround or a metalink note.
    Thanks in advance!

    Thanks for the fast reply!
    - Yes, the shared storage is accessible.
    - The alert log for the +ASM2 clearly shows that ASM instance has started normally using default parameters and at one point PMON process dumped.
    - The system logs just shows that there is an error executing "crswrapexece.pl"
    System Log
    ===================
    *Oct 24 19:25:03 host2 root: [ID 702911 user.error] exec /u01/11.2.0/grid/perl/bin/perl -I/u01/11.2.0/grid/perl/lib /u01/11.2.0/grid/bin/crswrapexece.pl /*
    u01/11.2.0/grid/crs/install/s_crsconfig_host2_env.txt /u01/11.2.0/grid/bin/ohasd.bin "reboot"
    Oct 24 19:26:33 host2 oracleoks: [ID 902884 kern.notice] [Oracle OKS] mallocing log buffer, size=10485760
    Oct 24 19:26:33 host2 oracleoks: [ID 714332 kern.notice] [Oracle OKS] log buffer = 0x301780fcb50, size 10485760
    Oct 24 19:26:33 host2 oracleoks: [ID 400061 kern.notice] NOTICE: [Oracle OKS] ODLM hash size 16384
    Oct 24 19:26:33 host2 oracleoks: [ID 160659 kern.notice] NOTICE: OKSK-00004: Module load succeeded. Build information: (LOW DEBUG) USM_11.2.0.3.0_SOLAR
    IS.SPARC64_110803.1 2011/08/11 02:38:30
    Oct 24 19:26:33 host2 pseudo: [ID 129642 kern.info] pseudo-device: oracleadvm0
    Oct 24 19:26:33 host2 genunix: [ID 936769 kern.info] oracleadvm0 is /pseudo/oracleadvm@0
    Oct 24 19:26:33 host2 oracleoks: [ID 141287 kern.notice] NOTICE: ADVMK-00001: Module load succeeded. Build information: (LOW DEBUG) - USM_11.2.0.3.0_SOL
    ARIS.SPARC64_110803.1 built on 2011/08/11 02:40:17.
    Oct 24 19:26:33 host2 oracleacfs: [ID 202941 kern.notice] NOTICE: [Oracle ACFS] FCB hash size 16384
    Oct 24 19:26:33 host2 oracleacfs: [ID 671725 kern.notice] NOTICE: [Oracle ACFS] buffer cache size 511MB (79884 buckets)
    Oct 24 19:26:33 host2 oracleacfs: [ID 730054 kern.notice] NOTICE: [Oracle ACFS] DLM hash size 16384
    Oct 24 19:26:33 host2 oracleoks: [ID 617314 kern.notice] NOTICE: ACFSK-0037: Module load succeeded. Build information: (LOW DEBUG) USM_11.2.0.3.0_SOLAR
    IS.SPARC64_110803.1 2011/08/11 02:42:45
    Oct 24 19:26:33 host2 pseudo: [ID 129642 kern.info] pseudo-device: oracleacfs0
    Oct 24 19:26:33 host2 genunix: [ID 936769 kern.info] oracleacfs0 is /pseudo/oracleacfs@0
    Oct 24 19:26:36 host2 oracleoks: [ID 621795 kern.notice] NOTICE: OKSK-00010: Persistent OKS log opened at /u01/11.2.0/grid/log/host2/acfs/acfs.log.0.
    Oct 24 19:31:37 host2 last message repeated 1 time
    Oct 24 19:33:05 host2 CLSD: [ID 770310 daemon.notice] The clock on host host2 has been updated by the Cluster Time Synchronization Service to be synchr
    onous with the mean cluster time.
    ASM alert log
    ====================================================================
    <msg time='2011-10-24T19:35:48.776+01:00' org_id='oracle' comp_id='asm'
    client_id='' type='UNKNOWN' level='16'
    host_id='host2' host_addr='10.172.16.200' module=''
    pid='26406'>
    <txt>System state dump requested by (instance=2, osid=26396 (PMON)), summary=[abnormal instance termination].
    </txt>
    </msg>
    <msg time='2011-10-24T19:35:48.778+01:00' org_id='oracle' comp_id='asm'
    client_id='' type='UNKNOWN' level='16'
    host_id='host2' host_addr='10.172.16.200' module=''
    pid='26406'>
    <txt>System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_26406.trc
    </txt>
    </msg>
    <msg time='2011-10-24T19:35:48.927+01:00' org_id='oracle' comp_id='asm'
    type='UNKNOWN' level='16' host_id='host2'
    host_addr='10.172.16.200' pid='26470'>
    <txt>ORA-1092 : opitsk aborting process
    </txt>
    </msg>
    <msg time='2011-10-24T19:35:49.128+01:00' org_id='oracle' comp_id='asm'
    type='UNKNOWN' level='16' host_id='host2'
    host_addr='10.172.16.200' pid='26472'>
    <txt>ORA-1092 : opitsk aborting process
    </txt>
    </msg>
    Output from "/u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_26406.trc"
    REQUEST:system state dump at level 10, requested by (instance=2, osid=26396 (PMON)), summary=[abnormal instance termination].
    kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).
    ===================================================
    SYSTEM STATE (level=10)
    Orapids on dead process list: [count = 0]
    PROCESS 1:
    SO: 0x3df098b50, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df098b50, name=process, file=ksu.h LINE:12616 ID:, pg=0
    (process) Oracle pid:1, ser:0, calls cur/top: 0x0/0x0
    flags : (0x20) PSEUDO
    flags2: (0x0), flags3: (0x10)
    intr error: 0, call error: 0, sess error: 0, txn error 0
    intr queue: empty
    ksudlp FALSE at location: 0
    (post info) last post received: 0 0 0
    last post received-location: No post
    last process to post me: none
    last post sent: 0 0 0
    last post sent-location: No post
    last process posted by me: none
    (latch info) wait_event=0 bits=0
    O/S info: user: , term: , ospid: (DEAD)
    OSD pid info: Unix process pid: 0, image: PSEUDO
    SO: 0x38000cef0, type: 5, owner: 0x3df098b50, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x0, name=kss parent, file=kss2.h LINE:138 ID:, pg=0
    PSO child state object changes :
    Dump of memory from 0x00000003DF722AC0 to 0x00000003DF722CC8
    3DF722AC0 00000000 00000000 00000000 00000000 [................]
    Repeat 31 times
    3DF722CC0 00000000 00000000 [........]
    PROCESS 2: PMON
    SO: 0x3df099bf8, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df099bf8, name=process, file=ksu.h LINE:12616 ID:, pg=0
    (process) Oracle pid:2, ser:1, calls cur/top: 0x3db6c8d30/0x3db6c8d30
    flags : (0xe) SYSTEM
    flags2: (0x0), flags3: (0x10)
    intr error: 0, call error: 0, sess error: 0, txn error 0
    intr queue: empty
    ksudlp FALSE at location: 0
    (post info) last post received: 0 0 136
    last post received-location: kjm.h LINE:1228 ID:kjmdmi: pmon to attach
    last process to post me: 3df0a2138 1 6
    last post sent: 0 0 137
    last post sent-location: kjm.h LINE:1230 ID:kjiath: pmon attached
    last process posted by me: 3df0a2138 1 6
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: 0x3debbbf40
    O/S info: user: grid, term: UNKNOWN, ospid: 26396
    OSD pid info: Unix process pid: 26396, image: oracle@host2 (PMON)
    SO: 0x3d8800c18, type: 30, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df099bf8, name=ges process, file=kji.h LINE:3669 ID:, pg=0
    GES MSG BUFFERS: st=emp chunk=0x0 hdr=0x0 lnk=0x0 flags=0x0 inc=0
    outq=0 sndq=0 opid=0 prmb=0x0
    mbg=(0 0) mbg=(0 0) mbg[r]=(0 0)
    fmq=(0 0) fmq=(0 0) fmq[r]=(0 0)
    mop[s]=0 mop[q]=0 pendq=0 zmbq=0
    nonksxp_recvs=0
    ------------process 3d8800c18--------------------
    proc version : 0
    Local inst : 2
    pid : 26396
    lkp_inst : 2
    svr_mode : 0
    proc state : KJP_FROZEN
    Last drm hb acked : 0
    flags : x50
    ast_rcvd_svrmod : 0
    current lock op : 0
    Total accesses : 1
    Imm. accesses : 0
    Locks on ASTQ : 0
    Locks Pending AST : 0
    Granted locks : 0
    AST_Q:
    PENDING_Q:
    GRANTED_Q:
    SO: 0x3d9835198, type: 14, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
    proc=0x3df099bf8, name=channel handle, file=ksr2.h LINE:367 ID:, pg=0
    (broadcast handle) 3d9835198 flag: (2) ACTIVE SUBSCRIBER,
    owner: 3df099bf8 - ospid: 26396
    event: 1, last message event: 1,
    last message waited event: 1,
    next message: 0(0), messages read: 0
    channel: (3d9934df8) PMON actions channel [name: 2]
    scope: 7, event: 1, last mesage event: 0,
    publishers/subscribers: 0/1,
    messages published: 0
    heuristic msg queue length: 0
    SO: 0x3d9835008, type: 14, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
    proc=0x3df099bf8, name=channel handle, file=ksr2.h LINE:367 ID:, pg=0
    (broadcast handle) 3d9835008 flag: (2) ACTIVE SUBSCRIBER,
    owner: 3df099bf8 - ospid: 26396
    event: 1, last message event: 1,
    last message waited event: 1,
    next message: 0(0), messages read: 0
    channel: (3d9941e40) scumnt mount lock [name: 157]
    scope: 1, event: 12, last mesage event: 0,
    publishers/subscribers: 0/12,
    messages published: 0
    heuristic msg queue length: 0
    SO: 0x3de4a2b80, type: 4, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df099bf8, name=session, file=ksu.h LINE:12624 ID:, pg=0
    (session) sid: 33 ser: 1 trans: 0x0, creator: 0x3df099bf8
    flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
    flags2: (0x409) -/-/INC
    DID: , short-term DID:
    txn branch: 0x0
    oct: 0, prv: 0, sql: 0x0, psql: 0x0, user: 0/SYS
    ksuxds FALSE at location: 0
    service name: SYS$BACKGROUND
    Current Wait Stack:
    Not in wait; last wait ended 0.666415 sec ago
    Wait State:
    fixed_waits=0 flags=0x21 boundary=0x0/-1
    Session Wait History:
    elapsed time of 0.666593 sec since last wait
    0: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=63 seq_num=64 snap_id=1
    wait times: snap=3.000089 sec, exc=3.000089 sec, total=3.000089 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.002067 sec of elapsed time
    1: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=62 seq_num=63 snap_id=1
    wait times: snap=3.010111 sec, exc=3.010111 sec, total=3.010111 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.001926 sec of elapsed time
    2: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=61 seq_num=62 snap_id=1
    wait times: snap=3.125286 sec, exc=3.125286 sec, total=3.125286 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.003361 sec of elapsed time
    3: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=60 seq_num=61 snap_id=1
    wait times: snap=3.000081 sec, exc=3.000081 sec, total=3.000081 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.002102 sec of elapsed time
    4: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0

  • 11gR2 RAC install fail when running root.sh script on second node

    I get the errors:
    ORA-15018: diskgroup cannot be created
    ORA-15072: command requires at least 2 regular failure groups, discovered only 0
    ORA-15080: synchronous I/O operation to a disk failed
    [main] [ 2012-04-10 16:44:12.564 EDT ] [UsmcaLogger.logException:175] oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-15018: diskgroup cannot be created
    ORA-15072: command requires at least 2 regular failure groups, discovered only 0
    ORA-15080: synchronous I/O operation to a disk failed
    I have tried the fix solutions from metalink note, but did not fix issue
    11GR2 GRID INFRASTRUCTURE INSTALLATION FAILS WHEN RUNNING ROOT.SH ON NODE 2 OF RAC USING ASMLIB [ID 1059847.1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

    Hi,
    it looks like, that your "shared device" you are using is not really shared.
    The second node does "create an ASM diskgroup" and create OCR and Voting disks. If this indeed would be a shared device, he should have recognized, that your disk is shared.
    So as a result your VMware configuration must be wrong, and the disk you presented as shared disk is not really shared.
    Which VMWare version did you use? It will not work correctly with the workstation or player edition, since shared disks are only really working with the server version.
    If you indeed using the server, could you paste your vm configurations?
    Furthermore I recommend using Virtual Box. There is a nice how-to:
    http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnOEL5UsingVirtualBox.php
    Sebastian

Maybe you are looking for