During the installation of grid infra(cluster) for Oracle 11.2 RAC one.

Question

Good Day All, and thanks in advance…
During the installation of grid infrastructure(cluster) for Oracle 11.2 RAC One Node on AIX6.1 ( PROD) , ASM used. I am getting below errors when executing ./root.sh
Upon investigation ,I managed to get note: 1068212.1 from the support oracle site ( see below for details) . I might be hitting Unpublished bug 8670579. I also logged Severity 2 SR with Oracle support to get the bug/patch fix and no one has attended the call.
This might be configuration issue or otherwise , if you have experienced the same issue please assist ? ( if you need more logfiles please feel free to request)….
I ran the Cluster Verify Check – all passed.
Many Thanks
Ezekiel Filane
/u01/app/11.2.0/grid#./root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-10-19 10:33:11: Parsing the host name
2010-10-19 10:33:11: Checking for super user privileges
2010-10-19 10:33:11: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User grid has the required capabilities to run CSSD in realtime mode
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'csgipm'
CRS-2672: Attempting to start 'ora.mdnsd' on 'csgipm'
CRS-2676: Start of 'ora.gipcd' on 'csgipm' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'csgipm' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'csgipm'
CRS-2676: Start of 'ora.gpnpd' on 'csgipm' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'csgipm'
CRS-2676: Start of 'ora.cssdmonitor' on 'csgipm' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'csgipm'
CRS-2672: Attempting to start 'ora.diskmon' on 'csgipm'
CRS-2676: Start of 'ora.diskmon' on 'csgipm' succeeded
CRS-2676: Start of 'ora.cssd' on 'csgipm' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'csgipm'
Start action for daemon aborted
CRS-2674: Start of 'ora.ctssd' on 'csgipm' failed
CRS-2679: Attempting to clean 'ora.ctssd' on 'csgipm'
CRS-2681: Clean of 'ora.ctssd' on 'csgipm' succeeded
CRS-4000: Command Start failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.ctssd -init
Start of resource "ora.ctssd -init" failed
Clusterware exclusive mode start of resource ora.ctssd failed
CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
CRS-4000: Command Stop failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
Stop of resource "ora.crsd -init" failed
Failed to stop CRSD
CRS-2500: Cannot stop resource 'ora.asm' as it is not running
CRS-4000: Command Stop failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.asm -init
Stop of resource "ora.asm -init" failed
Failed to stop ASM
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'csgipm'
CRS-2677: Stop of 'ora.cssdmonitor' on 'csgipm' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'csgipm'
CRS-2677: Stop of 'ora.cssd' on 'csgipm' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'csgipm'
CRS-2677: Stop of 'ora.gpnpd' on 'csgipm' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'csgipm'
CRS-2677: Stop of 'ora.gipcd' on 'csgipm' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'csgipm'
CRS-2677: Stop of 'ora.mdnsd' on 'csgipm' succeeded
Initial cluster configuration failed. See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_csgipm.log for details
csgipm:/u01/app/11.2.0/grid#ps -ef | grep pmon
root 6160492 3932160 0 10:54:13 pts/2 0:00 grep pmon
more /u01/app/11.2.0/grid/log/csgipm/client/ocrconfig_5767204.log
csgipm:/usr/sbin#more /u01/app/11.2.0/grid/log/csgipm/client/ocrconfig_5767204.log
2010-10-19 10:33:14.435: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 4
2010-10-19 10:33:14.435: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 5
2010-10-19 10:33:14.435: [  OCRRAW][1]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
2010-10-19 10:33:14.435: [  OCRRAW][1]proprioini: all disks are not OCR/OLR formatted
2010-10-19 10:33:14.435: [  OCRRAW][1]proprinit: Could not open raw device
2010-10-19 10:33:14.442: [ default][1]a_init:7!: Backend init unsuccessful : [26]
2010-10-19 10:33:14.461: [ OCRCONF][1]Exporting OCR data to [OCRUPGRADEFILE]
2010-10-19 10:33:14.461: [  OCRAPI][1]a_init:7!: Backend init unsuccessful : [33]
2010-10-19 10:33:14.461: [ OCRCONF][1]There was no previous version of OCR. error:[PROCL-33: Oracle Local Registry is not configured]
2010-10-19 10:33:14.461: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 0
2010-10-19 10:33:14.461: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 1
2010-10-19 10:33:14.462: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 2
2010-10-19 10:33:14.462: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 3
2010-10-19 10:33:14.462: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 4
2010-10-19 10:33:14.462: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 5
2010-10-19 10:33:14.462: [  OCRRAW][1]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
2010-10-19 10:33:14.462: [  OCRRAW][1]proprioini: all disks are not OCR/OLR formatted
2010-10-19 10:33:14.462: [  OCRRAW][1]proprinit: Could not open raw device
2010-10-19 10:33:14.462: [ default][1]a_init:7!: Backend init unsuccessful : [26]
2010-10-19 10:33:14.462: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 0
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 1
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 2
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 3
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 4
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 5
2010-10-19 10:33:14.463: [  OCRRAW][1]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 0
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 1
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 2
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 3
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 4
2010-10-19 10:33:14.463: [  OCROSD][1]utread:3: Problem reading buffer 104ef000 buflen 4096 retval 0 phy_offset 102400 retry 5
2010-10-19 10:33:14.483: [  OCRRAW][1]ibctx: Failed to read the whole bootblock. Assumes invalid format.
2010-10-19 10:33:14.483: [  OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22
2010-10-19 10:33:14.483: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 0
2010-10-19 10:33:14.483: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 1
2010-10-19 10:33:14.483: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 2
2010-10-19 10:33:14.484: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 3
2010-10-19 10:33:14.484: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 4
2010-10-19 10:33:14.484: [  OCROSD][1]utread:3: Problem reading buffer 104fe000 buflen 4096 retval 0 phy_offset 102400 retry 5
2010-10-19 10:33:14.484: [  OCRRAW][1]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
2010-10-19 10:33:14.541: [  OCRAPI][1]a_init:6a: Backend init successful
2010-10-19 10:33:14.646: [ OCRCONF][1]Initialized DATABASE keys
2010-10-19 10:33:14.650: [ OCRCONF][1]Exiting [status=success]...

Answer 1

Hi,
We are also trying to install 11.2.0.2 Grid infrastructure for Oracle RAC One Node on AIX 6.1. We did a POC in our lab environment and after much struggle got that working. Now we are building 4 clusters in the production environment and the first cluster installation failed while running root.sh on node2. We already have a Sev1 ticket open with Oracle Support but have not heard anything.
Here is root.sh output from node2. The two node names are p01dou416 and p01dou417.
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node p01dou416, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Failed to start Oracle Clusterware stack
Failed to start Cluster Synchorinisation Service in clustered mode at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1020.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
[root@P01DOU417] /u01/app/11.2.0/grid #
LOG output: /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/ rootcrs_p01dou417.log
2010-11-13 17:22:14: Successfully started requested Oracle stack daemons
2010-11-13 17:22:14: Starting CSS in clustered mode
2010-11-13 17:22:14: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl start resource ora.cssd -init
2010-11-13 17:32:28: Command output:
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'p01dou417'
CRS-2672: Attempting to start 'ora.gipcd' on 'p01dou417'
CRS-2676: Start of 'ora.cssdmonitor' on 'p01dou417' succeeded
CRS-2676: Start of 'ora.gipcd' on 'p01dou417' succeeded> CRS-2679: Attempting to clean 'ora.cssd' on 'p01dou417'
CRS-2681: Clean of 'ora.cssd' on 'p01dou417' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'p01dou417'
CRS-2677: Stop of 'ora.diskmon' on 'p01dou417' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'p01dou417'
CRS-2677: Stop of 'ora.gipcd' on 'p01dou417' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'p01dou417'
CRS-2677: Stop of 'ora.cssdmonitor' on 'p01dou417' succeeded
CRS-5804: Communication error with agent process
CRS-4000: Command Start failed, or completed with errors.
End Command output2010-11-13 17:32:28: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl check css
2010-11-13 17:32:28: Command output:
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
End Command output2010-11-13 17:32:28: Checking the status of css
2010-11-13 17:32:33: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl check css
2010-11-13 17:32:33: Command output:
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
End Command output2010-11-13 17:32:33: Checking the status of css
2010-11-13 17:32:38: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'p01dou417'
2010-11-13 17:32:38: CRS-2672: Attempting to start 'ora.gipcd' on 'p01dou417'
2010-11-13 17:32:38: CRS-2676: Start of 'ora.cssdmonitor' on 'p01dou417' succeeded
2010-11-13 17:32:38: CRS-2676: Start of 'ora.gipcd' on 'p01dou417' succeeded
2010-11-13 17:32:38: CRS-2672: Attempting to start 'ora.cssd' on 'p01dou417'
2010-11-13 17:32:38: CRS-2672: Attempting to start 'ora.diskmon' on 'p01dou417'
2010-11-13 17:32:38: CRS-2676: Start of 'ora.diskmon' on 'p01dou417' succeeded
2010-11-13 17:32:38: CRS-2674: Start of 'ora.cssd' on 'p01dou417' failed
2010-11-13 17:32:38: CRS-2679: Attempting to clean 'ora.cssd' on 'p01dou417'
2010-11-13 17:32:38: CRS-2681: Clean of 'ora.cssd' on 'p01dou417' succeeded
2010-11-13 17:32:38: CRS-2673: Attempting to stop 'ora.diskmon' on 'p01dou417'
2010-11-13 17:32:38: CRS-2677: Stop of 'ora.diskmon' on 'p01dou417' succeeded
2010-11-13 17:32:38: CRS-2673: Attempting to stop 'ora.gipcd' on 'p01dou417'
2010-11-13 17:32:38: CRS-2677: Stop of 'ora.gipcd' on 'p01dou417' succeeded
2010-11-13 17:32:38: CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'p01dou417'
2010-11-13 17:32:38: CRS-2677: Stop of 'ora.cssdmonitor' on 'p01dou417' succeeded
2010-11-13 17:32:38: CRS-5804: Communication error with agent process
2010-11-13 17:32:38: CRS-4000: Command Start failed, or completed with errors.
2010-11-13 17:32:38: Failed to start Oracle Clusterware stack
2010-11-13 17:32:38: ###### Begin DIE Stack Trace ######
2010-11-13 17:32:38: Package File Line Calling
2010-11-13 17:32:38: --------------- -------------------- ---- ----------
2010-11-13 17:32:38: 1: main rootcrs.pl 324 crsconfig_lib::dietrap
2010-11-13 17:32:38: 2: crsconfig_lib crsconfig_lib.pm 1020 main::__ANON__
2010-11-13 17:32:38: 3: crsconfig_lib crsconfig_lib.pm 997 crsconfig_lib::start_cluster
2010-11-13 17:32:38: 4: main rootcrs.pl 697 crsconfig_lib::perform_start_cluster
2010-11-13 17:32:38: ####### End DIE Stack Trace #######
2010-11-13 17:32:38: 'ROOTCRS_STACK' checkpoint has failed
Any help on this is appreciated.
Edited by: user12019257 on Nov 17, 2010 1:26 PM