Command return code of 255 (65280) during Grid Infrastructure Installation
I am trying to create a 2-node RAC 11gR2. I am using OEL 5 Update 4. I am using NFS as shared storage. The time on both node is same. Both nodes have 4GB RAM each. Following is my IP settings from /etc/hosts on both nodes:
[oracle@vis ~]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
# Public
10.8.5.191 vis.awc.com vis
10.8.5.192 vsstest.awc.com vsstest
#Private
192.168.0.101 vis-priv.awc.com vis-priv
192.168.0.102 vsstest-priv.awc.com vsstest-priv
#Virtual
10.8.5.166 vis-vip.awc.com vis-vip
10.8.5.170 vsstest-vip.awc.com vsstest-vip
# SCAN
10.8.5.182 rac-scan.awc.com rac-scan
#NAS
192.168.0.101 nas1.awc.com nas1
I am getting error during the Grid Infrastructure installation, when I run the root.sh script through root. Following is the logfile showing the error (/u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_vis.log):
2010-02-02 11:58:05: The configuration parameter file /u01/app/11.2.0/grid/crs/install/crsconfig_params is valid
2010-02-02 11:58:05: Checking for super user privileges
2010-02-02 11:58:05: User has super user privileges
2010-02-02 11:58:05: ### Printing the configuration values from files:
2010-02-02 11:58:05: /u01/app/11.2.0/grid/crs/install/crsconfig_params
2010-02-02 11:58:05: /u01/app/11.2.0/grid/crs/install/s_crsconfig_defs
2010-02-02 11:58:05: ASM_DISCOVERY_STRING=
2010-02-02 11:58:05: ASM_DISKS=
2010-02-02 11:58:05: ASM_DISK_GROUP=
2010-02-02 11:58:05: ASM_REDUNDANCY=
2010-02-02 11:58:05: ASM_SPFILE=
2010-02-02 11:58:05: ASM_UPGRADE=false
2010-02-02 11:58:05: CLSCFG_MISSCOUNT=
2010-02-02 11:58:05: CLUSTER_GUID=
2010-02-02 11:58:05: CLUSTER_NAME=rac-scan
2010-02-02 11:58:05: CRS_NODEVIPS='vis-vip/255.248.0.0/eth0,vsstest-vip/255.248.0.0/eth0'
2010-02-02 11:58:05: CRS_STORAGE_OPTION=2
2010-02-02 11:58:05: CSS_LEASEDURATION=400
2010-02-02 11:58:05: DIRPREFIX=
2010-02-02 11:58:05: DISABLE_OPROCD=0
2010-02-02 11:58:05: EMBASEJAR_NAME=oemlt.jar
2010-02-02 11:58:05: EWTJAR_NAME=ewt3.jar
2010-02-02 11:58:05: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
2010-02-02 11:58:05: GNS_ADDR_LIST=
2010-02-02 11:58:05: GNS_ALLOW_NET_LIST=
2010-02-02 11:58:05: GNS_CONF=false
2010-02-02 11:58:05: GNS_DENY_ITF_LIST=
2010-02-02 11:58:05: GNS_DENY_NET_LIST=
2010-02-02 11:58:05: GNS_DOMAIN_LIST=
2010-02-02 11:58:05: GPNPCONFIGDIR=/u01/app/11.2.0/grid
2010-02-02 11:58:05: GPNPGCONFIGDIR=/u01/app/11.2.0/grid
2010-02-02 11:58:05: GPNP_PA=
2010-02-02 11:58:05: HELPJAR_NAME=help4.jar
2010-02-02 11:58:05: HOST_NAME_LIST=vis,vsstest
2010-02-02 11:58:05: ID=/etc/init.d
2010-02-02 11:58:05: INIT=/sbin/init
2010-02-02 11:58:05: IT=/etc/inittab
2010-02-02 11:58:05: JEWTJAR_NAME=jewt4.jar
2010-02-02 11:58:05: JLIBDIR=/u01/app/11.2.0/grid/jlib
2010-02-02 11:58:05: JREDIR=/u01/app/11.2.0/grid/jdk/jre/
2010-02-02 11:58:06: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
2010-02-02 11:58:06: MSGFILE=/var/adm/messages
2010-02-02 11:58:06: NETCFGJAR_NAME=netcfg.jar
2010-02-02 11:58:06: NETWORKS="eth0"/10.8.0.0:public,"eth1"/192.168.0.0:cluster_interconnect
2010-02-02 11:58:06: NEW_HOST_NAME_LIST=
2010-02-02 11:58:06: NEW_NODEVIPS='vis-vip/255.248.0.0/eth0,vsstest-vip/255.248.0.0/eth0'
2010-02-02 11:58:06: NEW_NODE_NAME_LIST=
2010-02-02 11:58:06: NEW_PRIVATE_NAME_LIST=
2010-02-02 11:58:06: NODELIST=vis,vsstest
2010-02-02 11:58:06: NODE_NAME_LIST=vis,vsstest
2010-02-02 11:58:06: OCFS_CONFIG=
2010-02-02 11:58:06: OCRCONFIG=/etc/oracle/ocr.loc
2010-02-02 11:58:06: OCRCONFIGDIR=/etc/oracle
2010-02-02 11:58:06: OCRID=
2010-02-02 11:58:06: OCRLOC=ocr.loc
2010-02-02 11:58:06: OCR_LOCATIONS=/u01/shared_config/ocr_configuration
2010-02-02 11:58:06: OLASTGASPDIR=/etc/oracle/lastgasp
2010-02-02 11:58:06: OLRCONFIG=/etc/oracle/olr.loc
2010-02-02 11:58:06: OLRCONFIGDIR=/etc/oracle
2010-02-02 11:58:06: OLRLOC=olr.loc
2010-02-02 11:58:06: OPROCDCHECKDIR=/etc/oracle/oprocd/check
2010-02-02 11:58:06: OPROCDDIR=/etc/oracle/oprocd
2010-02-02 11:58:06: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
2010-02-02 11:58:06: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
2010-02-02 11:58:06: ORACLE_BASE=/u01/app/oracle
2010-02-02 11:58:06: ORACLE_HOME=/u01/app/11.2.0/grid
2010-02-02 11:58:06: ORACLE_OWNER=oracle
2010-02-02 11:58:06: ORA_ASM_GROUP=dba
2010-02-02 11:58:06: ORA_DBA_GROUP=oinstall
2010-02-02 11:58:06: PRIVATE_NAME_LIST=
2010-02-02 11:58:06: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
2010-02-02 11:58:06: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
2010-02-02 11:58:06: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
2010-02-02 11:58:06: RC_KILL=K19
2010-02-02 11:58:06: RC_KILL_OLD=K96
2010-02-02 11:58:06: RC_START=S96
2010-02-02 11:58:06: SCAN_NAME=rac-scan.awc.com
2010-02-02 11:58:06: SCAN_PORT=1521
2010-02-02 11:58:06: SCRBASE=/etc/oracle/scls_scr
2010-02-02 11:58:06: SHAREJAR_NAME=share.jar
2010-02-02 11:58:06: SILENT=false
2010-02-02 11:58:06: SO_EXT=so
2010-02-02 11:58:06: SRVCFGLOC=srvConfig.loc
2010-02-02 11:58:06: SRVCONFIG=/var/opt/oracle/srvConfig.loc
2010-02-02 11:58:06: SRVCONFIGDIR=/var/opt/oracle
2010-02-02 11:58:06: VNDR_CLUSTER=false
2010-02-02 11:58:06: VOTING_DISKS=/u01/shared_config/voting_disk
2010-02-02 11:58:06: ### Printing other configuration values ###
2010-02-02 11:58:06: CLSCFG_EXTRA_PARMS=
2010-02-02 11:58:06: CRSDelete=0
2010-02-02 11:58:06: CRSPatch=0
2010-02-02 11:58:06: DEBUG=
2010-02-02 11:58:06: DOWNGRADE=
2010-02-02 11:58:06: HAS_GROUP=oinstall
2010-02-02 11:58:06: HAS_USER=root
2010-02-02 11:58:06: HOST=vis
2010-02-02 11:58:07: IS_SIHA=0
2010-02-02 11:58:07: OLR_DIRECTORY=/u01/app/11.2.0/grid/cdata
2010-02-02 11:58:07: OLR_LOCATION=/u01/app/11.2.0/grid/cdata/vis.olr
2010-02-02 11:58:07: ORA_CRS_HOME=/u01/app/11.2.0/grid
2010-02-02 11:58:07: SUPERUSER=root
2010-02-02 11:58:07: UPGRADE=
2010-02-02 11:58:07: VF_DISCOVERY_STRING=/u01/shared_config/voting_disk
2010-02-02 11:58:07: addfile=/u01/app/11.2.0/grid/crs/install/crsconfig_addparams
2010-02-02 11:58:07: crscfg_trace=1
2010-02-02 11:58:07: crscfg_trace_file=/u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_vis.log
2010-02-02 11:58:07: hosts=
2010-02-02 11:58:07: oldcrshome=
2010-02-02 11:58:07: oldcrsver=
2010-02-02 11:58:07: osdfile=/u01/app/11.2.0/grid/crs/install/s_crsconfig_defs
2010-02-02 11:58:07: parameters_valid=1
2010-02-02 11:58:07: paramfile=/u01/app/11.2.0/grid/crs/install/crsconfig_params
2010-02-02 11:58:07: platform_family=unix
2010-02-02 11:58:07: srvctl_trc_suff=0
2010-02-02 11:58:07: unlock_crshome=
2010-02-02 11:58:07: user_is_superuser=1
2010-02-02 11:58:07: ### Printing of configuration values complete ###
2010-02-02 11:58:07: Oracle CRS stack is not configured yet
2010-02-02 11:58:07: CRS is not yet configured. Hence, will proceed to configure CRS
2010-02-02 11:58:07: Cluster-wide one-time actions... Done!
2010-02-02 11:58:17: set owner/group of OCR path
2010-02-02 11:58:17: Oracle CRS home = /u01/app/11.2.0/grid
2010-02-02 11:58:17: Host name = vis
2010-02-02 11:58:17: CRS user = oracle
2010-02-02 11:58:17: Oracle CRS home = /u01/app/11.2.0/grid
2010-02-02 11:58:17: GPnP host = vis
2010-02-02 11:58:17: Oracle GPnP home = /u01/app/11.2.0/grid/gpnp
2010-02-02 11:58:17: Oracle GPnP local home = /u01/app/11.2.0/grid/gpnp/vis
2010-02-02 11:58:17: GPnP directories verified.
2010-02-02 11:58:17: Checking to see if Oracle CRS stack is already configured
2010-02-02 11:58:17: Oracle CRS stack is not configured yet
2010-02-02 11:58:17: ---Checking local gpnp setup...
2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" does not exist
2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cwallet.sso" does not exist
2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/vis/wallets/prdr/cwallet.sso" does not exist
2010-02-02 11:58:18: chk gpnphome /u01/app/11.2.0/grid/gpnp/vis: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
2010-02-02 11:58:18: chk gpnphome /u01/app/11.2.0/grid/gpnp/vis: INVALID (bad profile/wallet)
2010-02-02 11:58:18: ---Checking cluster-wide gpnp setup...
2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/profiles/peer/profile.xml" does not exist
2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/wallets/peer/cwallet.sso" does not exist
2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/wallets/prdr/cwallet.sso" does not exist
2010-02-02 11:58:18: chk gpnphome /u01/app/11.2.0/grid/gpnp: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
2010-02-02 11:58:18: chk gpnphome /u01/app/11.2.0/grid/gpnp: INVALID (bad profile/wallet)
2010-02-02 11:58:18: gpnp setup checked: local valid? 0 cluster-wide valid? 0
2010-02-02 11:58:18: gpnp setup: NONE
2010-02-02 11:58:18: GPNP configuration required
2010-02-02 11:58:18: Validating for SI-CSS configuration
2010-02-02 11:58:18: Retrieving OCR main disk location
2010-02-02 11:58:18: Opening file OCRCONFIG
2010-02-02 11:58:18: Value () is set for key=ocrconfig_loc
2010-02-02 11:58:18: Unable to retrieve ocr disk info
2010-02-02 11:58:18: Checking to see if any 9i GSD is up
2010-02-02 11:58:18: libskgxnBase_lib = /etc/ORCLcluster/oracm/lib/libskgxn2.so
2010-02-02 11:58:18: libskgxn_lib = /opt/ORCLcluster/lib/libskgxn2.so
2010-02-02 11:58:18: SKGXN library file does not exists
2010-02-02 11:58:18: OLR location = /u01/app/11.2.0/grid/cdata/vis.olr
2010-02-02 11:58:18: Oracle CRS Home = /u01/app/11.2.0/grid
2010-02-02 11:58:18: Validating /etc/oracle/olr.loc file for OLR location /u01/app/11.2.0/grid/cdata/vis.olr
2010-02-02 11:58:18: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig
2010-02-02 11:58:18: Oracle CRS home = /u01/app/11.2.0/grid
2010-02-02 11:58:18: Oracle cluster name = rac-scan
2010-02-02 11:58:18: OCR locations = /u01/shared_config/ocr_configuration
2010-02-02 11:58:18: Validating OCR
2010-02-02 11:58:18: Retrieving OCR location used by previous installations
2010-02-02 11:58:18: Opening file OCRCONFIG
2010-02-02 11:58:18: Value () is set for key=ocrconfig_loc
2010-02-02 11:58:18: Opening file OCRCONFIG
2010-02-02 11:58:18: Value () is set for key=ocrmirrorconfig_loc
2010-02-02 11:58:18: Opening file OCRCONFIG
2010-02-02 11:58:18: Value () is set for key=ocrconfig_loc3
2010-02-02 11:58:18: Opening file OCRCONFIG
2010-02-02 11:58:18: Value () is set for key=ocrconfig_loc4
2010-02-02 11:58:18: Opening file OCRCONFIG
2010-02-02 11:58:18: Value () is set for key=ocrconfig_loc5
2010-02-02 11:58:18: Checking if OCR sync file exists
2010-02-02 11:58:18: No need to sync OCR file
2010-02-02 11:58:18: OCR_LOCATION=/u01/shared_config/ocr_configuration
2010-02-02 11:58:18: OCR_MIRROR_LOCATION=
2010-02-02 11:58:18: OCR_MIRROR_LOC3=
2010-02-02 11:58:18: OCR_MIRROR_LOC4=
2010-02-02 11:58:19: OCR_MIRROR_LOC5=
2010-02-02 11:58:19: Current OCR location=
2010-02-02 11:58:19: Current OCR mirror location=
2010-02-02 11:58:19: Current OCR mirror loc3=
2010-02-02 11:58:19: Current OCR mirror loc4=
2010-02-02 11:58:19: Current OCR mirror loc5=
2010-02-02 11:58:19: Verifying current OCR settings with user entered values
2010-02-02 11:58:19: Setting OCR locations in /etc/oracle/ocr.loc
2010-02-02 11:58:19: Validating OCR locations in /etc/oracle/ocr.loc
2010-02-02 11:58:19: Checking for existence of /etc/oracle/ocr.loc
2010-02-02 11:58:19: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig
2010-02-02 11:58:19: Setting ocr location /u01/shared_config/ocr_configuration
2010-02-02 11:58:19: Creating or upgrading Oracle Local Registry (OLR)
2010-02-02 11:58:20: OLR successfully created or upgraded
2010-02-02 11:58:20: /u01/app/11.2.0/grid/bin/clscfg -localadd
2010-02-02 11:58:21: Keys created in the OLR successfully
2010-02-02 11:58:21: GPnP setup state: none
2010-02-02 11:58:21: Creating local GPnP setup for clustered node...
2010-02-02 11:58:21: Oracle CRS home = /u01/app/11.2.0/grid
2010-02-02 11:58:21: Oracle GPnP wallets home = /u01/app/11.2.0/grid/gpnp/vis/wallets
2010-02-02 11:58:21: Checking if GPnP setup exists
2010-02-02 11:58:21: /u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cwallet.sso wallet must be created
2010-02-02 11:58:21: Removing old wallets/certificates, if any
2010-02-02 11:58:21: Creating GPnP Root Wallet...
2010-02-02 11:58:21: /u01/app/11.2.0/grid/bin/orapki wallet create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/root" -pwd gpnp_wallet1 -nologo
2010-02-02 11:58:22: Creating GPnP Root Certificate...
2010-02-02 11:58:22: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/root" -pwd gpnp_wallet1 -self_signed -dn "CN=GPnP_root" -keysize 1024 -validity 9999 -nologo
2010-02-02 11:58:24: Exporting GPnP Root Certificate...
2010-02-02 11:58:24: /u01/app/11.2.0/grid/bin/orapki wallet export -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/root" -pwd gpnp_wallet1 -dn "CN=GPnP_root" -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/root/b64certificate.txt" -nologo
2010-02-02 11:58:25: Creating GPnP Peer Wallet...
2010-02-02 11:58:25: /u01/app/11.2.0/grid/bin/orapki wallet create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -auto_login -nologo
2010-02-02 11:58:25: Creating GPnP Profile Reader Wallet...
2010-02-02 11:58:25: /u01/app/11.2.0/grid/bin/orapki wallet create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/prdr" -pwd gpnp_wallet1 -auto_login -nologo
2010-02-02 11:58:26: Creating GPnP PA Wallet...
2010-02-02 11:58:26: /u01/app/11.2.0/grid/bin/orapki wallet create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -auto_login -nologo
2010-02-02 11:58:27: Adding private key to GPnP Peer Wallet...
2010-02-02 11:58:27: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -dn "CN=GPnP_peer" -keysize 1024 -nologo
2010-02-02 11:58:28: Adding private key to GPnP PA Wallet...
2010-02-02 11:58:28: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -dn "CN=GPnP_pa" -keysize 1024 -nologo
2010-02-02 11:58:29: Creating certificate request for GPnP Peer Wallet...
2010-02-02 11:58:29: /u01/app/11.2.0/grid/bin/orapki wallet export -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -dn "CN=GPnP_peer" -request "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/certreq.txt" -nologo
2010-02-02 11:58:30: Creating certificate request for GPnP PA Wallet...
2010-02-02 11:58:30: /u01/app/11.2.0/grid/bin/orapki wallet export -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -dn "CN=GPnP_pa" -request "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/certreq.txt" -nologo
2010-02-02 11:58:31: Creating certificate for GPnP Peer Wallet...
2010-02-02 11:58:31: /u01/app/11.2.0/grid/bin/orapki cert create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/root" -pwd gpnp_wallet1 -request "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/certreq.txt" -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cert.txt" -validity 9999 -nologo
2010-02-02 11:58:32: Creating certificate for GPnP PA Wallet...
2010-02-02 11:58:32: /u01/app/11.2.0/grid/bin/orapki cert create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/root" -pwd gpnp_wallet1 -request "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/certreq.txt" -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/cert.txt" -validity 9999 -nologo
2010-02-02 11:58:33: Adding Root Certificate TP to GPnP Peer Wallet...
2010-02-02 11:58:33: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/root/b64certificate.txt" -nologo
2010-02-02 11:58:34: Adding Root Certificate TP to GPnP Profile Reader Wallet...
2010-02-02 11:58:34: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/prdr" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/root/b64certificate.txt" -nologo
2010-02-02 11:58:34: Adding Root Certificate TP to GPnP PA Wallet...
2010-02-02 11:58:34: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/root/b64certificate.txt" -nologo
2010-02-02 11:58:35: Adding PA Certificate as a TP into a GPnP Peer Wallet...
2010-02-02 11:58:35: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/cert.txt" -nologo
2010-02-02 11:58:36: Adding peer Certificate as a TP into a GPnP PA Wallet...
2010-02-02 11:58:36: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cert.txt" -nologo
2010-02-02 11:58:37: Adding PA Certificate as a TP into a GPnP Profile Reader Wallet...
2010-02-02 11:58:37: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/prdr" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/cert.txt" -nologo
2010-02-02 11:58:38: Adding peer Certificate as a TP into a GPnP Profile Reader Wallet...
2010-02-02 11:58:38: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/prdr" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cert.txt" -nologo
2010-02-02 11:58:39: Adding PA Certificate as a TP into a GPnP Peer Wallet...
2010-02-02 11:58:39: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -user_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cert.txt" -nologo
2010-02-02 11:58:40: Adding peer Certificate as a TP into a GPnP PA Wallet...
2010-02-02 11:58:40: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -user_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/cert.txt" -nologo
2010-02-02 11:58:41: GPnP Wallets ownership/permissions successfully set.
2010-02-02 11:58:41: GPnP Wallets successfully created.
2010-02-02 11:58:41: <--- GPnP wallets successfully created
2010-02-02 11:58:41: Creating GPnP peer profile --->
2010-02-02 11:58:41: Oracle CRS home = /u01/app/11.2.0/grid
2010-02-02 11:58:41: Oracle GPnP profiles home = /u01/app/11.2.0/grid/gpnp/vis/profiles
2010-02-02 11:58:41: Oracle GPnP profiles parameters:
2010-02-02 11:58:41: paloc==
2010-02-02 11:58:41: cname=rac-scan=
2010-02-02 11:58:41: cssdisco=/u01/shared_config/voting_disk=
2010-02-02 11:58:41: cssld=400=
2010-02-02 11:58:41: asmdisco==
2010-02-02 11:58:41: asmspf==
2010-02-02 11:58:41: netlst="eth0"/10.8.0.0:public,"eth1"/192.168.0.0:cluster_interconnect=
2010-02-02 11:58:41: ocrid==
2010-02-02 11:58:41: clusterguid==
2010-02-02 11:58:41: Checking if GPnP setup exists
2010-02-02 11:58:41: /u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml profile must be created
2010-02-02 11:58:41: OCRID is not available, hence not set in GPnP Profile
2010-02-02 11:58:42: ClusterGUID is not available, hence not set in GPnP Profile
2010-02-02 11:58:42: gpnptool: run /u01/app/11.2.0/grid/bin/gpnptool create -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -prf -prf_sq=1 -prf_cn=rac-scan -prf_pa="" -hnet=gen -gen:hnet_nm="*" -gen:net=net1 -net1:net_ip="10.8.0.0" -net1:net_ada="eth0" -net1:net_use="public" -gen:net=net2 -net2:net_ip="192.168.0.0" -net2:net_ada="eth1" -net2:net_use="cluster_interconnect" -css=css -css:css_dis="/u01/shared_config/voting_disk" -css:css_ld=400 -asm=asm -asm:asm_dis="++no-value-at-profile-creation--never-updated-through-ASM++" -asm:asm_spf=""
2010-02-02 11:58:42: Running as user oracle: /u01/app/11.2.0/grid/bin/gpnptool create -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -prf -prf_sq=1 -prf_cn=rac-scan -prf_pa="" -hnet=gen -gen:hnet_nm="*" -gen:net=net1 -net1:net_ip="10.8.0.0" -net1:net_ada="eth0" -net1:net_use="public" -gen:net=net2 -net2:net_ip="192.168.0.0" -net2:net_ada="eth1" -net2:net_use="cluster_interconnect" -css=css -css:css_dis="/u01/shared_config/voting_disk" -css:css_ld=400 -asm=asm -asm:asm_dis="++no-value-at-profile-creation--never-updated-through-ASM++" -asm:asm_spf=""
2010-02-02 11:58:42: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/11.2.0/grid/bin/gpnptool create -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -prf -prf_sq=1 -prf_cn=rac-scan -prf_pa="" -hnet=gen -gen:hnet_nm="*" -gen:net=net1 -net1:net_ip="10.8.0.0" -net1:net_ada="eth0" -net1:net_use="public" -gen:net=net2 -net2:net_ip="192.168.0.0" -net2:net_ada="eth1" -net2:net_use="cluster_interconnect" -css=css -css:css_dis="/u01/shared_config/voting_disk" -css:css_ld=400 -asm=asm -asm:asm_dis="++no-value-at-profile-creation--never-updated-through-ASM++" -asm:asm_spf="" '
2010-02-02 11:58:42: Removing file /tmp/fileXUKYkK
2010-02-02 11:58:42: Successfully removed file: /tmp/fileXUKYkK
2010-02-02 11:58:42: /bin/su successfully executed
2010-02-02 11:58:42: gpnptool: rc=0
2010-02-02 11:58:42: gpnptool output:
Resulting profile written to "/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml".
Success.
2010-02-02 11:58:42: gpnptool: run /u01/app/11.2.0/grid/bin/gpnptool sign -p="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -w="file:/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -rmws
2010-02-02 11:58:42: Running as user oracle: /u01/app/11.2.0/grid/bin/gpnptool sign -p="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -w="file:/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -rmws
2010-02-02 11:58:42: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/11.2.0/grid/bin/gpnptool sign -p="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -w="file:/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -rmws '
2010-02-02 11:58:43: Removing file /tmp/file4E7Vnj
2010-02-02 11:58:43: Successfully removed file: /tmp/file4E7Vnj
2010-02-02 11:58:43: /bin/su successfully executed
2010-02-02 11:58:43: gpnptool: rc=0
2010-02-02 11:58:43: gpnptool output:
Resulting profile written to "/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml".
Success.
2010-02-02 11:58:43: GPnP peer profile create successfully completed.
2010-02-02 11:58:43: <--- GPnP peer profile successfully created
2010-02-02 11:58:43: GPnP local setup successfully created
2010-02-02 11:58:43: Registering ohasd
2010-02-02 11:58:43: init file = /u01/app/11.2.0/grid/crs/init/init.ohasd
2010-02-02 11:58:43: Copying file /u01/app/11.2.0/grid/crs/init/init.ohasd to /etc/init.d directory
2010-02-02 11:58:43: Setting init.ohasd permission in /etc/init.d directory
2010-02-02 11:58:43: init file = /u01/app/11.2.0/grid/crs/init/ohasd
2010-02-02 11:58:43: Copying file /u01/app/11.2.0/grid/crs/init/ohasd to /etc/init.d directory
2010-02-02 11:58:43: Setting ohasd permission in /etc/init.d directory
2010-02-02 11:58:43: Removing "/etc/rc.d/rc3.d/S96ohasd"
2010-02-02 11:58:43: Removing file /etc/rc.d/rc3.d/S96ohasd
2010-02-02 11:58:43: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd
2010-02-02 11:58:43: Failed to remove file:
2010-02-02 11:58:43: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
2010-02-02 11:58:43: Removing "/etc/rc.d/rc5.d/S96ohasd"
2010-02-02 11:58:43: Removing file /etc/rc.d/rc5.d/S96ohasd
2010-02-02 11:58:43: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd
2010-02-02 11:58:43: Failed to remove file:
2010-02-02 11:58:43: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
2010-02-02 11:58:43: Removing "/etc/rc.d/rc0.d/K19ohasd"
2010-02-02 11:58:43: Removing file /etc/rc.d/rc0.d/K19ohasd
2010-02-02 11:58:43: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd
2010-02-02 11:58:43: Failed to remove file:
2010-02-02 11:58:43: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd
2010-02-02 11:58:43: Removing "/etc/rc.d/rc1.d/K19ohasd"
2010-02-02 11:58:43: Removing file /etc/rc.d/rc1.d/K19ohasd
2010-02-02 11:58:43: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd
2010-02-02 11:58:44: Failed to remove file:
2010-02-02 11:58:44: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd
2010-02-02 11:58:44: Removing "/etc/rc.d/rc2.d/K19ohasd"
2010-02-02 11:58:44: Removing file /etc/rc.d/rc2.d/K19ohasd
2010-02-02 11:58:44: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd
2010-02-02 11:58:44: Failed to remove file:
2010-02-02 11:58:44: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd
2010-02-02 11:58:44: Removing "/etc/rc.d/rc4.d/K19ohasd"
2010-02-02 11:58:44: Removing file /etc/rc.d/rc4.d/K19ohasd
2010-02-02 11:58:44: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd
2010-02-02 11:58:44: Failed to remove file:
2010-02-02 11:58:44: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd
2010-02-02 11:58:44: Removing "/etc/rc.d/rc6.d/K19ohasd"
2010-02-02 11:58:44: Removing file /etc/rc.d/rc6.d/K19ohasd
2010-02-02 11:58:44: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
2010-02-02 11:58:44: Failed to remove file:
2010-02-02 11:58:44: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
2010-02-02 11:58:44: The file ohasd has been successfully linked to the RC directories
2010-02-02 11:58:44: Starting ohasd
2010-02-02 11:58:44: itab entries=
2010-02-02 11:58:49: Created backup /etc/inittab.no_crs
2010-02-02 11:58:49: Appending to /etc/inittab.tmp:
2010-02-02 11:58:50: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
2010-02-02 11:58:50: Done updating /etc/inittab.tmp
2010-02-02 11:58:50: Saved /etc/inittab.crs
2010-02-02 11:58:50: Installed new /etc/inittab
2010-02-02 11:58:55: ohasd is starting
2010-02-02 11:58:55: Checking ohasd
2010-02-02 11:58:55: ohasd started successfully
2010-02-02 11:58:55: Creating CRS resources and dependencies
2010-02-02 11:58:55: Configuring HASD
2010-02-02 11:58:55: Registering type ora.daemon.type
2010-02-02 11:58:59: Registering type ora.mdns.type
2010-02-02 11:59:00: Registering type ora.gpnp.type
2010-02-02 11:59:01: Registering type ora.gipc.type
2010-02-02 11:59:02: Registering type ora.cssd.type
2010-02-02 11:59:04: Registering type ora.cssdmonitor.type
2010-02-02 11:59:07: Registering type ora.crs.type
2010-02-02 11:59:08: Registering type ora.evm.type
2010-02-02 11:59:09: Registering type ora.ctss.type
2010-02-02 11:59:10: Registering type ora.asm.type
2010-02-02 11:59:13: Registering type ora.drivers.acfs.type
2010-02-02 11:59:14: Registering type ora.diskmon.type
2010-02-02 11:59:34: ADVM/ACFS is configured
2010-02-02 11:59:35: Successfully created CRS resources for cluster daemon and ASM
2010-02-02 11:59:35: Checking if initial configuration has been performed
2010-02-02 11:59:35: Starting CSS in exclusive mode
2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.gipcd' on 'vis'
2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.mdnsd' on 'vis'
2010-02-02 12:00:04: CRS-2676: Start of 'ora.gipcd' on 'vis' succeeded
2010-02-02 12:00:04: CRS-2676: Start of 'ora.mdnsd' on 'vis' succeeded
2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.gpnpd' on 'vis'
2010-02-02 12:00:04: CRS-2676: Start of 'ora.gpnpd' on 'vis' succeeded
2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'vis'
2010-02-02 12:00:04: CRS-2676: Start of 'ora.cssdmonitor' on 'vis' succeeded
2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.cssd' on 'vis'
2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.diskmon' on 'vis'
2010-02-02 12:00:04: CRS-2676: Start of 'ora.diskmon' on 'vis' succeeded
2010-02-02 12:00:04: CRS-2676: Start of 'ora.cssd' on 'vis' succeeded
2010-02-02 12:00:04: Querying for existing CSS voting disks
2010-02-02 12:00:04: Performing initial configuration for cluster
2010-02-02 12:00:51: Start of resource "ora.ctssd -init" Succeeded
2010-02-02 12:00:51: Creating or upgrading OCR keys
2010-02-02 12:00:51: Command return code of 255 (65280) from command: /u01/app/11.2.0/grid/bin/ocrconfig -upgrade oracle oinstall
2010-02-02 12:00:51: Failed to create Oracle Cluster Registry configuration, rc 255
2010-02-02 12:00:51: Exiting exclusive mode
2010-02-02 12:00:51: Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
2010-02-02 12:00:51: Stop of resource "ora.crsd -init" failed
2010-02-02 12:00:51: Failed to stop CRSD
2010-02-02 12:01:20: Initial cluster configuration failed. See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_vis.log for details
Hi,
I am trying to setup Oracle 11gR2 RAC setup on RHEL 5.4 64-bit machine, during grid installation i am also getting the same error.
2010-07-12 03:12:42: Removing file /etc/rc.d/rc6.d/K19ohasd
2010-07-12 03:12:42: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
2010-07-12 03:12:42: Failed to remove file:
2010-07-12 03:12:42: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
2010-07-12 03:12:42: The file ohasd has been successfully linked to the RC directories
2010-07-12 03:12:42: Starting ohasd
2010-07-12 03:12:42: itab entries=
2010-07-12 03:12:47: Created backup /etc/inittab.no_crs
2010-07-12 03:12:47: Appending to /etc/inittab.tmp:
2010-07-12 03:12:47: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
2010-07-12 03:12:47: Done updating /etc/inittab.tmp
2010-07-12 03:12:47: Saved /etc/inittab.crs
2010-07-12 03:12:47: Installed new /etc/inittab
2010-07-12 03:13:03: ohasd is starting
2010-07-12 03:13:03: Checking ohasd
2010-07-12 03:13:03: ohasd started successfully
2010-07-12 03:13:03: Creating CRS resources and dependencies
2010-07-12 03:13:03: Configuring HASD
2010-07-12 03:13:03: Registering type ora.daemon.type
2010-07-12 03:13:04: Registering type ora.mdns.type
2010-07-12 03:13:04: Registering type ora.gpnp.type
2010-07-12 03:13:04: Registering type ora.gipc.type
2010-07-12 03:13:05: Registering type ora.cssd.type
2010-07-12 03:13:05: Registering type ora.cssdmonitor.type
2010-07-12 03:13:06: Registering type ora.crs.type
2010-07-12 03:13:06: Registering type ora.evm.type
2010-07-12 03:13:07: Registering type ora.ctss.type
2010-07-12 03:13:07: Registering type ora.asm.type
2010-07-12 03:13:08: Registering type ora.drivers.acfs.type
2010-07-12 03:13:08: Registering type ora.diskmon.type
2010-07-12 03:13:35: ADVM/ACFS is configured
2010-07-12 03:13:35: Successfully created CRS resources for cluster daemon and ASM
2010-07-12 03:13:35: Checking if initial configuration has been performed
2010-07-12 03:13:35: Starting CSS in exclusive mode
2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.gipcd' on 'lodvmrhn3'
2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.mdnsd' on 'lodvmrhn3'
2010-07-12 03:14:04: CRS-2676: Start of 'ora.mdnsd' on 'lodvmrhn3' succeeded
2010-07-12 03:14:04: CRS-2676: Start of 'ora.gipcd' on 'lodvmrhn3' succeeded
2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.gpnpd' on 'lodvmrhn3'
2010-07-12 03:14:04: CRS-2676: Start of 'ora.gpnpd' on 'lodvmrhn3' succeeded
2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lodvmrhn3'
2010-07-12 03:14:04: CRS-2676: Start of 'ora.cssdmonitor' on 'lodvmrhn3' succeeded
2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.cssd' on 'lodvmrhn3'
2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.diskmon' on 'lodvmrhn3'
2010-07-12 03:14:04: CRS-2676: Start of 'ora.diskmon' on 'lodvmrhn3' succeeded
2010-07-12 03:14:04: CRS-2676: Start of 'ora.cssd' on 'lodvmrhn3' succeeded
2010-07-12 03:14:04: Querying for existing CSS voting disks
2010-07-12 03:14:05: Performing initial configuration for cluster
2010-07-12 03:14:06: Start of resource "ora.ctssd -init" Succeeded
2010-07-12 03:14:06: Creating or upgrading OCR keys
2010-07-12 03:14:06: Command return code of 255 (65280) from command: /u01/app/11.2.0/grid/bin/ocrconfig -upgrade oracle oinstall
2010-07-12 03:14:06: Failed to create Oracle Cluster Registry configuration, rc 255
2010-07-12 03:14:06: Exiting exclusive mode
2010-07-12 03:14:06: Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
2010-07-12 03:14:06: Stop of resource "ora.crsd -init" failed
2010-07-12 03:14:06: Failed to stop CRSD
2010-07-12 03:14:34: Initial cluster configuration failed. See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_lodvmrhn3.log for details
Please hel me to resole the issue.
I have followed exact steps mentioned in the http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnLinuxUsingNFS.php
Similar Messages
-
11g R2 RAC - Grid Infrastructure installation - "root.sh" fails on node#2
Hi there,
I am trying to create a two node 11g R2 RAC on OEL 5.5 (32-bit) using VMWare virtual machines. I have correctly configured both nodes. Cluster Verification utility returns on following error \[which I believe can be ignored]:
Checking daemon liveness...
Liveness check failed for "ntpd"
Check failed on nodes:
rac2,rac1
PRVF-5415 : Check to see if NTP daemon is running failed
Clock synchronization check using Network Time Protocol(NTP) failed
Pre-check for cluster services setup was unsuccessful on all the nodes.
While Grid Infrastructure installation (for a Cluster option), things go very smooth until I run "root.sh" on node# 2. orainstRoot.sh ran OK on both. "root.sh" run OK on node# 1 and ends with:
Checking swap space: must be greater than 500 MB. Actual 1967 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
*'UpdateNodeList' was successful.*
*[root@rac1 ~]#*
"root.sh" fails on rac2 (2nd node) with following error:
CRS-2672: Attempting to start 'ora.evmd' on 'rac2'
CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded
Timed out waiting for the CRS stack to start.
*[root@rac2 ~]#*
I know this info may not be enough to figure out what the problem may be. Please let me know what should I look for to find the issue and fix it. Its been like almost two weeks now :-(
Regards
AmerHi Zheng,
ocssd.log is HUGE. So I am putting few of the last lines in the log file hoping they may give some clue:
2011-07-04 19:49:24.007: [ CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 2180 > margin 1500 cur_ms 36118424 lastalive 36116244
2011-07-04 19:49:26.005: [ CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 4150 > margin 1500 cur_ms 36120424 lastalive 36116274
2011-07-04 19:49:26.006: [ CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 4180 > margin 1500 cur_ms 36120424 lastalive 36116244
2011-07-04 19:49:27.997: [ CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:49:27.997: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:49:33.001: [ CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:49:33.001: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:49:37.996: [ CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:49:37.996: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:49:43.000: [ CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:49:43.000: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:49:48.004: [ CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:49:48.005: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:12.003: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:12.008: [ CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1660 > margin 1500 cur_ms 36166424 lastalive 36164764
2011-07-04 19:50:12.009: [ CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1660 > margin 1500 cur_ms 36166424 lastalive 36164764
2011-07-04 19:50:15.796: [ CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 2130 > margin 1500 cur_ms 36170214 lastalive 36168084
2011-07-04 19:50:16.996: [ CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:50:16.996: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:17.826: [ CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1540 > margin 1500 cur_ms 36172244 lastalive 36170704
2011-07-04 19:50:17.826: [ CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1570 > margin 1500 cur_ms 36172244 lastalive 36170674
2011-07-04 19:50:21.999: [ CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:50:21.999: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:26.011: [ CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1740 > margin 1500 cur_ms 36180424 lastalive 36178684
2011-07-04 19:50:26.011: [ CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1620 > margin 1500 cur_ms 36180424 lastalive 36178804
2011-07-04 19:50:27.004: [ CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:50:27.004: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:28.002: [ CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1700 > margin 1500 cur_ms 36182414 lastalive 36180714
2011-07-04 19:50:28.002: [ CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1790 > margin 1500 cur_ms 36182414 lastalive 36180624
2011-07-04 19:50:31.998: [ CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:50:31.998: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:37.001: [ CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:50:37.002: [ CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
*<end of log file>*And the alertrac2.log contains:
*[root@rac2 rac2]# cat alertrac2.log*
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2011-07-02 16:43:51.571
[client(16134)]CRS-2106:The OLR location /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olr is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/client/ocrconfig_16134.log.
2011-07-02 16:43:57.125
[client(16134)]CRS-2101:The OLR was formatted using version 3.
2011-07-02 16:44:43.214
[ohasd(16188)]CRS-2112:The OLR service started on node rac2.
2011-07-02 16:45:06.446
[ohasd(16188)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
2011-07-02 16:53:30.061
[ohasd(16188)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2011-07-02 16:53:55.042
[cssd(17674)]CRS-1713:CSSD daemon is started in exclusive mode
2011-07-02 16:54:38.334
[cssd(17674)]CRS-1707:Lease acquisition for node rac2 number 2 completed
[cssd(17674)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1 and is terminating; details at (:CSSNM00006:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log
2011-07-02 16:54:38.464
[cssd(17674)]CRS-1603:CSSD on node rac2 shutdown by user.
2011-07-02 16:54:39.174
[ohasd(16188)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac2'.
2011-07-02 16:55:43.430
[cssd(17945)]CRS-1713:CSSD daemon is started in clustered mode
2011-07-02 16:56:02.852
[cssd(17945)]CRS-1707:Lease acquisition for node rac2 number 2 completed
2011-07-02 16:56:04.061
[cssd(17945)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
2011-07-02 16:56:18.350
[cssd(17945)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2011-07-02 16:56:29.283
[ctssd(18020)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
2011-07-02 16:56:29.551
[ctssd(18020)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
2011-07-02 16:56:29.615
[ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 16:56:29.616
[ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 16:56:29.641
[ctssd(18020)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
[client(18052)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(18056)]CRS-10001:ACFS-9322: done.
2011-07-02 17:01:40.963
[ohasd(16188)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ohasd/ohasd.log.
[client(18590)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(18594)]CRS-10001:ACFS-9322: done.
2011-07-02 17:27:46.385
[ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 17:27:46.385
[ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 17:46:48.717
[crsd(22519)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:46:49.641
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:46:51.459
[crsd(22553)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:46:51.776
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:46:53.928
[crsd(22574)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:46:53.956
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:46:55.834
[crsd(22592)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:46:56.273
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:46:57.762
[crsd(22610)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:46:58.631
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:00.259
[crsd(22628)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:00.968
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:02.513
[crsd(22645)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:03.309
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:05.081
[crsd(22663)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:05.770
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:07.796
[crsd(22681)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:08.257
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:10.733
[crsd(22699)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:11.739
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:13.547
[crsd(22732)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:14.111
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:14.112
[ohasd(16188)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2011-07-02 17:58:18.459
[ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 17:58:18.459
[ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
[client(26883)]CRS-10001:ACFS-9200: Supported
2011-07-02 18:13:34.627
[ctssd(18020)]CRS-2405:The Cluster Time Synchronization Service on host rac2 is shutdown by user
2011-07-02 18:13:42.368
[cssd(17945)]CRS-1603:CSSD on node rac2 shutdown by user.
2011-07-02 18:15:13.877
[client(27222)]CRS-2106:The OLR location /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olr is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/client/ocrconfig_27222.log.
2011-07-02 18:15:14.011
[client(27222)]CRS-2101:The OLR was formatted using version 3.
2011-07-02 18:15:23.226
[ohasd(27261)]CRS-2112:The OLR service started on node rac2.
2011-07-02 18:15:23.688
[ohasd(27261)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2011-07-02 18:15:24.064
[ohasd(27261)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
2011-07-02 18:16:29.761
[ohasd(27261)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2011-07-02 18:16:30.190
[gpnpd(28498)]CRS-2328:GPNPD started on node rac2.
2011-07-02 18:16:41.561
[cssd(28562)]CRS-1713:CSSD daemon is started in exclusive mode
2011-07-02 18:16:49.111
[cssd(28562)]CRS-1707:Lease acquisition for node rac2 number 2 completed
2011-07-02 18:16:49.166
[cssd(28562)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
[cssd(28562)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1 and is terminating; details at (:CSSNM00006:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log
2011-07-02 18:17:01.122
[cssd(28562)]CRS-1603:CSSD on node rac2 shutdown by user.
2011-07-02 18:17:06.917
[ohasd(27261)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac2'.
2011-07-02 18:17:23.602
[mdnsd(28485)]CRS-5602:mDNS service stopping by request.
2011-07-02 18:17:36.217
[gpnpd(28732)]CRS-2328:GPNPD started on node rac2.
2011-07-02 18:17:43.673
[cssd(28794)]CRS-1713:CSSD daemon is started in clustered mode
2011-07-02 18:17:49.826
[cssd(28794)]CRS-1707:Lease acquisition for node rac2 number 2 completed
2011-07-02 18:17:49.865
[cssd(28794)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
2011-07-02 18:18:03.049
[cssd(28794)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2011-07-02 18:18:06.160
[ctssd(28861)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
2011-07-02 18:18:06.220
[ctssd(28861)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
2011-07-02 18:18:06.238
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 18:18:06.239
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 18:18:06.794
[ctssd(28861)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
[client(28891)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(28895)]CRS-10001:ACFS-9322: done.
2011-07-02 18:18:33.465
[crsd(29020)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:33.575
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:35.757
[crsd(29051)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:36.129
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:38.596
[crsd(29066)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:39.146
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:41.058
[crsd(29085)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:41.435
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:44.255
[crsd(29101)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:45.165
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:47.013
[crsd(29121)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:47.409
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:50.071
[crsd(29136)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:50.118
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:51.843
[crsd(29156)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:52.373
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:54.361
[crsd(29171)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:54.772
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:56.620
[crsd(29202)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:57.104
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:58.997
[crsd(29218)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:59.301
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:59.302
[ohasd(27261)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2011-07-02 18:49:58.070
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 18:49:58.070
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 19:21:33.362
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 19:21:33.362
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 19:52:05.271
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 19:52:05.271
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 20:22:53.696
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 20:22:53.696
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 20:53:43.949
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 20:53:43.949
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 21:24:32.990
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 21:24:32.990
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 21:55:21.907
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 21:55:21.908
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 22:26:45.752
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 22:26:45.752
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 22:57:54.682
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 22:57:54.683
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 23:07:28.603
[cssd(28794)]CRS-1612:Network communication with node rac1 (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.020 seconds
2011-07-02 23:07:35.621
[cssd(28794)]CRS-1611:Network communication with node rac1 (1) missing for 75% of timeout interval. Removal of this node from cluster in 7.010 seconds
2011-07-02 23:07:39.629
[cssd(28794)]CRS-1610:Network communication with node rac1 (1) missing for 90% of timeout interval. Removal of this node from cluster in 3.000 seconds
2011-07-02 23:07:42.641
[cssd(28794)]CRS-1632:Node rac1 is being removed from the cluster in cluster incarnation 205080558
2011-07-02 23:07:44.751
[cssd(28794)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
2011-07-02 23:07:45.326
[ctssd(28861)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
2011-07-04 19:46:26.008
[ohasd(27261)]CRS-8011:reboot advisory message from host: rac1, component: mo155738, with time stamp: L-2011-07-04-19:44:43.318
[ohasd(27261)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
*[root@rac2 rac2]#* This log file start with complaint that OLR is not accessible. Here is what I see (rca2):
-rw------- 1 root oinstall 272756736 Jul 2 18:18 /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olrAnd I guess rest of the problems start with this. -
Hello,
It may sound too easy to someone but I need to ask it anyway.
I am in the middle of building Oracle RAC 11.2.0.3 on Linux. I have 2 storage and I created three LUNs in each of storage so 6 LUNs in total look in both of servers. If I choose NORMAL as a redundancy level, is it right to choose my 6 disks those are created for OCR_VOTE at the grid installation? Or should I choose only 3 of them and configure mirroring at latter stage?
The reason why I am asking this is that I didn't find any place to create asm disk group for ocr and voting disks at the oracle grid infrastructure installation window. In fact, I would to like to create two disk groups in which one of groups contains three disks those were brought from storage 1 and the second one contains rest three 3 disk that come from the last storage.
I believe that you will understand the manner and help me on choosing proper way.
Thank you.Hi,
You have 2 Storage H/W.
You will need to configure a Quorum ASMDisk to store a Voting Disk.
Because if you lose 1/2 or more of all of your voting disks, then nodes get evicted from the cluster, or nodes kick themselves out of the cluster.
You must have the odd number of voting disk (i.e 1,3 or 5) one voting disk per ASM DISK, So 1 Storage will hold more voting disk than another storage.
(e.g 3 Voting Disk - 2 voting stored in stg-A and 1 voting disk store in stg-B)
If fail the storage that hold the major number of Voting disk the whole cluster (all nodes) goes down. Does not matter if you have a other storage online.
It will happen:
https://forums.oracle.com/message/9681811#9681811
You must configure your Clusterware same as RAC extended
Check this link:
http://www.oracle.com/technetwork/database/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf
Explaining: How to store OCR, Voting disks and ASM SPFILE on ASM Diskgroup (RAC or RAC Extended) | Levi Pereira | Oracl… -
11.2 Standalone Grid Infrastructure installation to Clustered Grid Insall
Quick question:
Currently we have a 11.2.0.1 standalone grid infrastructure installation with ASM instance running on server a1. And there are few databases running under the +ASM instance on a1. We are planning to make a1, a2 as a 2-node real application cluster. Is there a way to convert the standalone grid infrastructure installation on server a1 to a clustered grid infrastructure installation along with a2 with out de-installing the existing standalone grid infrastructure on a1 and re-installing freshly?
ThanksHi Boochi,
Thanks for responding. May be I am not clear enough in my post. Database conversion will be the next step, my question is about the actual grid infrastructure installation...Currently a1 server has been configured to work as standalone grid infrastructure, but our requirement is to convert to a 2-node RAC with server a2. Generally, I would go by de-installing the existing standalone grid infrastructure running on a1 and re-install the clustered-grid infrastructure with nodes a1 and a2. I am wondering if there is a way without de-installing the existing standalone grid infrastructure?? -
Blog Post: Grid Infrastructure installation for Standalone
I hope This document should be useful for who trying to install Grid infrastructure for Standalone server..., Expect comments Also.Read it Enjoy it.
http://mohamedazar.wordpress.com/2010/11/29/grid-infrastructure-installation-guide-for-standalone-server/Please check this metalink note:
OHASD Failed to Start: Inappropriate ioctl for device (Doc ID 1069182.1) -
Oracle Grid infrastructure installation tutorial
Dear friends who can help me by providing some link for Oracle Grid infrastructure installation
tutorial. I need installation procedure description with GNS, SCAN as well as DNS and DHCP
configuration for Grid Infra. I already searched in NET and can't found same exhaustive instructions.
I just want to compare my and others installations ways.
I hope see your comments soon.
thanks in advanceora_tech wrote:
Hi,
Refer blog post by Guenadi Jilevski:
http://gjilevski.wordpress.com/2011/10/05/build-two-node-oracle-rac-11gr2-11-2-0-3-with-gns-dns-dhcp-and-haip/
This might be helpful :)
thanks,
X A H E E Rthanks for quick answer :) -
Oracle grid infrastructure installation take long time on step 15 (perform prerequisite checks )
Oracle grid infrastructure installation take long time on step 15 (perform prerequisite checks )
I'm starting install and configure Oracle RAC 11gR2 on redhat 6.3 , when i reached step 15 (perform prerequisite checks ) it's taking long time more than 10 hours and still checking .... i closed it and start installation again it's same still waiting in perform prerequisite checks 75% complete
it is normal ?? what should i do ?? how i can monitoring installation steps and why this step take all this time please help .Pl identify which exact version of 11gR2 - currently only 11.2.0.3 is certified/supported on Linux 6.x - Oracle&reg; Database - if you are using 11.2.0.1 or 11.2.0.2, then you will need to use Linux 5.x
HTH
Srini -
DDIC ACTIVATION ERRORS and RETURN CODE in SAPA700VT1.LBP During Prepare
Hi Gurus,
For The Error mesage During prepare
Restart logs of DB conversions found
n Outstanding DB conversions found
We refer the Upgrade Guide & followed the steps thro SE14 Choose >DB Requests> Terminated .
Nothing found.
Then we Edit the Table found in error log ( CONV ENTRY TBATG INDX/BIC/FZLDEL_C11 0600CREB 200710032STEAM) . & by mistekan used the option Extras > Force conversion.
Now table TABTG is missing in from DATA Base & we are facing the Error
DDIC ACTIVATION ERRORS and RETURN CODE in SAPA700VT1.LBP
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 ETP111 exit code : "12"
Again when we check SE14 Choose >DB Requests> Terminated it shows one Entry TBATG but is not exist now
Help meHi,
Did you checked SM13 or SMQ1? that´s the probable cause for that error, now you should open an OSS note or start all over again. -
Root.sh failure on node1 during Grid Infra installation
I’m installing 11.2.0.3 Grid Infrastructure on two AIX 7.1 nodes. Root.sh script fails at ora.cssd startup. I’m using manual network configuration with DNS and hosts file. Cluvfy output looks good but root.sh fails. Any ideas?
Cluvfy output indicates overall passed minus some ignorable errors. Also, I verified interconnect and asm disks and they appear all good. I have some concernes about these (bellow), should I worry about them at this stage?
Network parameter - ipqmaxlen - Checks if the network parameter is set correctly on the system
Check Failed on Nodes: [donner02, donner01]
Verification result of failed node: donner02
Expected Value
: 512
Actual Value
: en1=
Details:
PRVE-0273 : The value of network parameter "ipqmaxlen" for interface "en1" is not configured to the expected value on node "donner02".[Expected="512"; Found="en1="] - Cause: - Action:
Back to Top
Verification result of failed node: donner01
Expected Value
: 512
Actual Value
: en1=
Details:
PRVE-0273 : The value of network parameter "ipqmaxlen" for interface "en1" is not configured to the expected value on node "donner01".[Expected="512"; Found="en1="] - Cause: - Action:
Network parameter - sb_max - Checks if the network parameter is set correctly on the system
Check Failed on Nodes: [donner02, donner01]
Verification result of failed node: donner02
Expected Value
: 4194304
Actual Value
: en1=
Details:
PRVE-0273 : The value of network parameter "sb_max" for interface "en1" is not configured to the expected value on node "donner02".[Expected="4194304"; Found="en1="] - Cause: - Action:
Back to Top
Verification result of failed node: donner01
Expected Value
: 4194304
Actual Value
: en1=
Details:
PRVE-0273 : The value of network parameter "sb_max" for interface "en1" is not configured to the expected value on node "donner01".[Expected="4194304"; Found="en1="] - Cause: - Action:
Network parameter - tcp_sendspace - Checks if the network parameter is set correctly on the system
Check Failed on Nodes: [donner02, donner01]
Verification result of failed node: donner02
Expected Value
: 65536
Actual Value
: en1=131072
Details:
PRVE-0273 : The value of network parameter "tcp_sendspace" for interface "en1" is not configured to the expected value on node "donner02".[Expected="65536"; Found="en1=131072"] - Cause: - Action:
Back to Top
Verification result of failed node: donner01
Expected Value
: 65536
Actual Value
: en1=131072
Details:
PRVE-0273 : The value of network parameter "tcp_sendspace" for interface "en1" is not configured to the expected value on node "donner01".[Expected="65536"; Found="en1=131072"] - Cause: - Action:
Network parameter - udp_sendspace - Checks if the network parameter is set correctly on the system
Check Failed on Nodes: [donner02, donner01]
Verification result of failed node: donner02
Expected Value
: 65536
Actual Value
: en1=
Details:
PRVE-0273 : The value of network parameter "udp_sendspace" for interface "en1" is not configured to the expected value on node "donner02".[Expected="65536"; Found="en1="] - Cause: - Action:
Back to Top
Verification result of failed node: donner01
Expected Value
: 65536
Actual Value
: en1=
Details:
PRVE-0273 : The value of network parameter "udp_sendspace" for interface "en1" is not configured to the expected value on node "donner01".[Expected="65536"; Found="en1="] - Cause: - Action:
Network parameter - udp_recvspace - Checks if the network parameter is set correctly on the system
Check Failed on Nodes: [donner02, donner01]
Verification result of failed node: donner02
Expected Value
: 655360
Actual Value
: en1=
Details:
PRVE-0273 : The value of network parameter "udp_recvspace" for interface "en1" is not configured to the expected value on node "donner02".[Expected="655360"; Found="en1="] - Cause: - Action:
Back to Top
Verification result of failed node: donner01
Expected Value
: 655360
Actual Value
: en1=
Details:
PRVE-0273 : The value of network parameter "udp_recvspace" for interface "en1" is not configured to the expected value on node "donner01".[Expected="655360"; Found="en1="] - Cause: - Action: -
What happens after root.sh - grid infrastructure installation
Hi,
I'm trying to install 11.2.0.2 grid infrastructure.
I got ok until root.sh, during root.sh I got some error on the asm disks for the ocr - I think it's a permissions issue.
Anyway - sysadmin fixed it and I know I now need to deconfigure root.sh -force and then run it root.sh again.
however - sysadmin also reboot my server so - the installer is GONE.
My question is - let say I get the root.sh run smothly, what is the installer doing after I press ok on the root.sh screen (which I now of course don't have...)
can this be run in some script as well?
Thanks!just to close this thread - the answer I gor from oracle support was:
Hello,
The OUI will execute cluvfy after root.sh is acknowledge to be run successfully. You can manually execute cluvfy from grid_home with -post option to validate the installation.
check also with commands that all resources are online:
crsctl stat res -init
crsctl stat res -t
Thanks and regards, -
PSU patching and Grid Infrastructure installation
GRID version: 11.2.0.3 , 11.2.0.2
Platform : AIX 6.1
Found a similair post by spiral in OTN on. But couldn't find the answer there; hence posting this
We install grid using Grid Installer GUI. When doing a fresh installation of Grid Infrastructure 11.2.0.2 or 11.2.0.3 in a new server , I want to apply its latest PSU as well. When should I do this PSU patching ?
A. Should I apply the GI PSU after the entire Grid installation is successfully completed ?
or
B. Should I apply the GI PSU just before running the root.sh ?
ie. when the installer prompts you to run root.sh (see the screenshot below)
http://husnusensoy.files.wordpress.com/2009/10/camtasiastudio351.png
At this stage(before running root.sh) the GI binaries are in place . So , I can apply the PSU on these binaries? Right?Should I follow option A or B above ?Hi,
A. Should I apply the GI PSU after the entire Grid installation is successfully completed ?Yes, You must finish installation sucessful and after apply PSU
B. Should I apply the GI PSU just before running the root.sh ?
ie. when the installer prompts you to run root.sh (see the screenshot below)
http://husnusensoy.files.wordpress.com/2009/10/camtasiastudio351.png
At this stage(before running root.sh) the GI binaries are in place . So , I can apply the PSU on these binaries? Right?Yes you should finish (close) runInstaller after root.sh, so you can apply PSU, because if you start opatch with OUI (runInstaller) running the Inventory will be locked.
Also you have a option to decide if you want to use the Software Updates option during Installation.
OUI can install critical patch updates, system requirements updates (hardware, operating system parameters, and kernel packages) for supported operating systems, and other significant updates that can help to ensure your installation proceeds smoothly. Oracle recommends that you enable software updates during installation.
If you choose to enable software updates, then during installation you must provide a valid My Oracle Support user name and password during installation, so that OUI can download the latest updates, or you must provide a path to the location of an software updates packages that you have downloaded previously.
If you plan to run the installation in a secured data center, then you can download updates before starting the installation by starting OUI on a system that has Internet access in update download mode. To start OUI to download updates, enter the following command:
$ ./runInstaller -downloadUpdatesProvide the My Oracle Support user name and password, and provide proxy settings if needed. After you download updates, transfer the update file to a directory on the server where you plan to run the installation.
Regards,
Levi Pereira -
Troubles with Grid Infrastructure Installation
Hello everybody
Enviroment:
O.S: SLES SP1 x64
Oracle 11g R2 Update 3
I want to install this product and make a RAC of tow nodes. Since some days ago I have a problem with the installation process. When I perform ssh connectivity setup, this errors appears at ssh log configuration:
INFO: [rac2.mydomain.com]
INFO: /bin/mkdir -p "/opt/oracle/.ssh"
INFO: Exit-status: 0
INFO: Error:
INFO:
INFO:
INFO: [rac2.mydomain.com]
INFO: /bin/rm -f "/opt/oracle/.ssh/id_rsa.pubrac2.mydomain.com"
INFO: Exit-status: 0
INFO: Error:
INFO:
INFO:
INFO: Copy localfile: /opt/oracle/.ssh/id_rsa.pub to remotefile: "/opt/oracle/.ssh/id_rsa.pubrac2.mydomain.com"
INFO: [rac2.mydomain.com]
INFO: /bin/cp -p "/opt/oracle/.ssh/authorized_keys" "/opt/oracle/.ssh/authorized_keys.ri.bak"
INFO: Exit-status: 1
INFO: Error: /bin/cp: cannot stat `/opt/oracle/.ssh/authorized_keys': No such file or directory
INFO: Creating keys on remote Host rac1.mydomain.com
INFO: to setup SSH on Host rac1.mydomain.com
INFO: [rac1.mydomain.com]
INFO: /bin/rm /opt/oracle/.ssh/identity.pub
INFO: Exit-status: 1
INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity.pub': No such file or directory
INFO:
INFO:
INFO: [rac1.mydomain.com]
INFO: /bin/rm /opt/oracle/.ssh/identity
INFO: Exit-status: 1
INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity': No such file or directory
NFO:
INFO: Creating keys on remote Host rac2.mydomain.com
INFO: to setup SSH on Host rac2.mydomain.com
INFO: [rac2.mydomain.com]
INFO: /bin/rm /opt/oracle/.ssh/identity.pub
INFO: Exit-status: 1
INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity.pub': No such file or directory
INFO:
INFO:
INFO: [rac2.mydomain.com]
INFO: /bin/rm /opt/oracle/.ssh/identity
INFO: Exit-status: 1
INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity': No such file or directory
At installation logs, these lines appear:
INFO: SSH Setup policy used: BIDIRECTIONAL
INFO: Preparing to check passwordless SSH Connectivity between nodes: [rac1, rac2]
INFO: Testing passwordless SSH connectivity between the selected nodes. This may take several minutes, please wait...
INFO: OverallStatus of User Equivalence check using CVU is SUCCESSFUL
INFO: Finding passwordless SSH connectivity setup on remote nodes
INFO: Preparing remote commands to check passwordless SSH connectivity setup on remote nodes
INFO: Total number of remote commands to be executed is 2
WARNING: Failed while executing remote commands to check SSH connectivity to other nodes..
Refer associated stacktrace #oracle.install.driver.oui.OUISSHSupportManager:887
INFO: Source Node: rac2 Target Node: rac1
INFO: Exception: null
INFO: Status log: oracle.install.driver.oui.OUISSHSupportManager$SSHSetupVerificationCommand: 0|Permission denied (publickey,keyboard-interac
tive). :failed
INFO: Source Node: rac2 Target Node: rac2
INFO: Exception: null
INFO: Status log: oracle.install.driver.oui.OUISSHSupportManager$SSHSetupVerificationCommand: 0|Permission denied (publickey,keyboard-interac
tive). :failed
INFO: Remote node(s) not configured for passwordless SSH connectivity : [rac2]
INFO: Details about the list of remote node(s), not configured for passwordless SSH connectivity
NODE REMOTE NODES
rac2 [rac1, rac2]
INFO: Pause Oracle Grid Infrastructure
INFO: Resume Oracle Grid Infrastructure
INFO: Adding ExitStatus USER_CANCELLED_INSTALL to the exit status set
INFO: Finding the most appropriate exit status for the current application
INFO: Exit Status is 4
INFO: Shutdown Oracle Grid Infrastructure
INFO: Unloading Setup Driver
To confirm that everything is going wrong, when I click in Test button, this error appears:
[INS_06006] Passwordless SSH connectivity not set up between the following node(s): [rac2]
I can do ssh from rac1 to rac2 without password, but I can´t from rac2 to rac1
I run cluvfy.sh and everything is ok.
I hope for helping.
Thanks a lotThe problem begins when on remote node It can´t perform a several operations as logs appears.
INFO: [rac2.mydomain.com]
INFO: /bin/cp -p "/opt/oracle/.ssh/authorized_keys" "/opt/oracle/.ssh/authorized_keys.ri.bak"
INFO: Exit-status: 1
INFO: Error: /bin/cp: cannot stat `/opt/oracle/.ssh/authorized_keys': No such file or directory
INFO: Creating keys on remote Host rac1.mydomain.com
INFO: to setup SSH on Host rac1.mydomain.com
INFO: [rac1.mydomain.com]
INFO: /bin/rm /opt/oracle/.ssh/identity.pub
INFO: Exit-status: 1
INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity.pub': No such file or directory
INFO:
INFO:
INFO: [rac1.mydomain.com]
INFO: /bin/rm /opt/oracle/.ssh/identity
INFO: Exit-status: 1
INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity': No such file or directory
NFO:
INFO: Creating keys on remote Host rac2.mydomain.com
INFO: to setup SSH on Host rac2.mydomain.com
INFO: [rac2.mydomain.com]
INFO: /bin/rm /opt/oracle/.ssh/identity.pub
INFO: Exit-status: 1
INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity.pub': No such file or directory
INFO:
INFO:
INFO: [rac2.mydomain.com]
INFO: /bin/rm /opt/oracle/.ssh/identity
INFO: Exit-status: 1
INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity': No such file or directory -
Grid Infrastructure installable for Oracle Restart only (No ASM)?
Hello,
Can Oracle Grid Infrastructure be installed for its Oracle Restart functionality only? i.e., No ASM,; this box uses local RAID 5 filesystems only.
This is a standalone 32-bit RHEL 5 system with Oracle Database 10gR2 already installed that I intend to upgrade to 11gR2. Just experimenting with installing Grid Infrastructure for only its restart capability. There isn't anything out there I've found that addresses this other than the Oracle doc stating that the Infrastructure must be installed if I want to use ASM or Oracle Restart.
I'm currently in the installer in a Software Only installation waiting for my SysAdmin to run the root.sh script, but in looking at this script, it looks like it might try to do things with CRS, voting files and the like, which do not exist.
Thank you!
-JonHi Erik,
Can Oracle Grid Infrastructure be installed for its Oracle Restart functionality only? Yes ... you can do that installing Software Only
I'm currently in the installer in a Software Only installation waiting for my SysAdmin to run the root.sh script, but in looking at this script, it looks like it might try to do things with CRS, voting files and the like, which do not exist.Don't worry about that ... at the end of the execution of the root.sh script will report a proceedings to configure only Oracle Restart.
Only this resource "ora.evmd" will appear online.
Regards,
Levi Pereira -
Oracle Grid Infrastructure installation on Windows 2008 SRV R2
Hi
There are many problems to configure and install grid infrastructure in windows 2008 server. is there any step by step documentation for this case?
how can i configure the storage for OCFS, ACFS is not supported on Windows 2008 server. How can i manage OUI setup masks exactly.
Thank you for your helping hand.Have you looked at the instructions in the documentation? http://download.oracle.com/docs/cd/E11882_01/install.112/e10817/toc.htm
If these steps are not adequate, could you explain what is missing? -
Grid Infrastructure installation file
Hi,
I am planning to install Grid Infrastructure for RAC 11.2.
Can someone please send me the correct link from where I can download the correct version?
ThanksThe thing is I am not able to view the grid installation options in my installer. I have the normal DB installer.
So trying to download the installer which has the grid installation options too.
Maybe you are looking for
-
Passing prompt value from one report to other using navigation
I have one dashboard with one prompt (Prompt Name: Date) and a report with date column. This date in the prompt defaults to current date by using server variable 'CURRENT_DATE'. When a user executes this dashboard the report will be executed for curr
-
Creative Zen 32 GB - disconnect when battery is f
Hello! I have bought the Creative Zen 32 GB and after i connected it to my USB-Port and loaded fully the first time, it abort the connection to the PC every few minutes and provokes the Automatic Execution window ( like described in this thread: "Cre
-
Jdev Design View not appropriate
My design view is not appropriate while i have just added search panel view criteria and table below it. Even for master child table things are not coming properly. Is this because of using bundles for text? Any help??? please....
-
Flash running extremely slow since today's update
After I updated I-tunes and the system this morning, flash animations in Safari (specifically Fishworld) run very very slowly. The same program causes Firefox to crash. Any ideas?
-
Passing array of values from front panel to menu
I am working with a group programming a multichannel current stimulation device and we are currently having trouble setting up the run time menu. The burning question that we have is trying to figure out how to pass an array of values from the front