11g R2 RAC - Grid Infrastructure installation - "root.sh" fails on node#2

Hi there,
I am trying to create a two node 11g R2 RAC on OEL 5.5 (32-bit) using VMWare virtual machines. I have correctly configured both nodes. Cluster Verification utility returns on following error \[which I believe can be ignored]:
Checking daemon liveness...
Liveness check failed for "ntpd"
Check failed on nodes:
rac2,rac1
PRVF-5415 : Check to see if NTP daemon is running failed
Clock synchronization check using Network Time Protocol(NTP) failed
Pre-check for cluster services setup was unsuccessful on all the nodes.
While Grid Infrastructure installation (for a Cluster option), things go very smooth until I run "root.sh" on node# 2. orainstRoot.sh ran OK on both. "root.sh" run OK on node# 1 and ends with:
Checking swap space: must be greater than 500 MB.   Actual 1967 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
*'UpdateNodeList' was successful.*
*[root@rac1 ~]#*
"root.sh" fails on rac2 (2nd node) with following error:
CRS-2672: Attempting to start 'ora.evmd' on 'rac2'
CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded
Timed out waiting for the CRS stack to start.
*[root@rac2 ~]#*
I know this info may not be enough to figure out what the problem may be. Please let me know what should I look for to find the issue and fix it. Its been like almost two weeks now :-(
Regards
Amer

Hi Zheng,
ocssd.log is HUGE. So I am putting few of the last lines in the log file hoping they may give some clue:
2011-07-04 19:49:24.007: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 2180 > margin 1500  cur_ms 36118424 lastalive 36116244
2011-07-04 19:49:26.005: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 4150 > margin 1500 cur_ms 36120424 lastalive 36116274
2011-07-04 19:49:26.006: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 4180 > margin 1500  cur_ms 36120424 lastalive 36116244
2011-07-04 19:49:27.997: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:49:27.997: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:49:33.001: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:49:33.001: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:49:37.996: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:49:37.996: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:49:43.000: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:49:43.000: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:49:48.004: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:49:48.005: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:12.003: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:12.008: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1660 > margin 1500 cur_ms 36166424 lastalive 36164764
2011-07-04 19:50:12.009: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1660 > margin 1500  cur_ms 36166424 lastalive 36164764
2011-07-04 19:50:15.796: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 2130 > margin 1500  cur_ms 36170214 lastalive 36168084
2011-07-04 19:50:16.996: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:50:16.996: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:17.826: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1540 > margin 1500 cur_ms 36172244 lastalive 36170704
2011-07-04 19:50:17.826: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1570 > margin 1500  cur_ms 36172244 lastalive 36170674
2011-07-04 19:50:21.999: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:50:21.999: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:26.011: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1740 > margin 1500 cur_ms 36180424 lastalive 36178684
2011-07-04 19:50:26.011: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1620 > margin 1500  cur_ms 36180424 lastalive 36178804
2011-07-04 19:50:27.004: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:50:27.004: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:28.002: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1700 > margin 1500 cur_ms 36182414 lastalive 36180714
2011-07-04 19:50:28.002: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1790 > margin 1500  cur_ms 36182414 lastalive 36180624
2011-07-04 19:50:31.998: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:50:31.998: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
2011-07-04 19:50:37.001: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
2011-07-04 19:50:37.002: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
*<end of log file>*And the alertrac2.log contains:
*[root@rac2 rac2]# cat alertrac2.log*
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2011-07-02 16:43:51.571
[client(16134)]CRS-2106:The OLR location /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olr is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/client/ocrconfig_16134.log.
2011-07-02 16:43:57.125
[client(16134)]CRS-2101:The OLR was formatted using version 3.
2011-07-02 16:44:43.214
[ohasd(16188)]CRS-2112:The OLR service started on node rac2.
2011-07-02 16:45:06.446
[ohasd(16188)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
2011-07-02 16:53:30.061
[ohasd(16188)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2011-07-02 16:53:55.042
[cssd(17674)]CRS-1713:CSSD daemon is started in exclusive mode
2011-07-02 16:54:38.334
[cssd(17674)]CRS-1707:Lease acquisition for node rac2 number 2 completed
[cssd(17674)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1 and is terminating; details at (:CSSNM00006:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log
2011-07-02 16:54:38.464
[cssd(17674)]CRS-1603:CSSD on node rac2 shutdown by user.
2011-07-02 16:54:39.174
[ohasd(16188)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac2'.
2011-07-02 16:55:43.430
[cssd(17945)]CRS-1713:CSSD daemon is started in clustered mode
2011-07-02 16:56:02.852
[cssd(17945)]CRS-1707:Lease acquisition for node rac2 number 2 completed
2011-07-02 16:56:04.061
[cssd(17945)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
2011-07-02 16:56:18.350
[cssd(17945)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2011-07-02 16:56:29.283
[ctssd(18020)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
2011-07-02 16:56:29.551
[ctssd(18020)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
2011-07-02 16:56:29.615
[ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 16:56:29.616
[ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 16:56:29.641
[ctssd(18020)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
[client(18052)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(18056)]CRS-10001:ACFS-9322: done.
2011-07-02 17:01:40.963
[ohasd(16188)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ohasd/ohasd.log.
[client(18590)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(18594)]CRS-10001:ACFS-9322: done.
2011-07-02 17:27:46.385
[ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 17:27:46.385
[ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 17:46:48.717
[crsd(22519)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:46:49.641
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:46:51.459
[crsd(22553)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:46:51.776
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:46:53.928
[crsd(22574)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:46:53.956
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:46:55.834
[crsd(22592)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:46:56.273
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:46:57.762
[crsd(22610)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:46:58.631
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:00.259
[crsd(22628)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:00.968
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:02.513
[crsd(22645)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:03.309
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:05.081
[crsd(22663)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:05.770
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:07.796
[crsd(22681)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:08.257
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:10.733
[crsd(22699)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:11.739
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:13.547
[crsd(22732)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 17:47:14.111
[ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 17:47:14.112
[ohasd(16188)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2011-07-02 17:58:18.459
[ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 17:58:18.459
[ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
[client(26883)]CRS-10001:ACFS-9200: Supported
2011-07-02 18:13:34.627
[ctssd(18020)]CRS-2405:The Cluster Time Synchronization Service on host rac2 is shutdown by user
2011-07-02 18:13:42.368
[cssd(17945)]CRS-1603:CSSD on node rac2 shutdown by user.
2011-07-02 18:15:13.877
[client(27222)]CRS-2106:The OLR location /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olr is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/client/ocrconfig_27222.log.
2011-07-02 18:15:14.011
[client(27222)]CRS-2101:The OLR was formatted using version 3.
2011-07-02 18:15:23.226
[ohasd(27261)]CRS-2112:The OLR service started on node rac2.
2011-07-02 18:15:23.688
[ohasd(27261)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2011-07-02 18:15:24.064
[ohasd(27261)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
2011-07-02 18:16:29.761
[ohasd(27261)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2011-07-02 18:16:30.190
[gpnpd(28498)]CRS-2328:GPNPD started on node rac2.
2011-07-02 18:16:41.561
[cssd(28562)]CRS-1713:CSSD daemon is started in exclusive mode
2011-07-02 18:16:49.111
[cssd(28562)]CRS-1707:Lease acquisition for node rac2 number 2 completed
2011-07-02 18:16:49.166
[cssd(28562)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
[cssd(28562)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1 and is terminating; details at (:CSSNM00006:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log
2011-07-02 18:17:01.122
[cssd(28562)]CRS-1603:CSSD on node rac2 shutdown by user.
2011-07-02 18:17:06.917
[ohasd(27261)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac2'.
2011-07-02 18:17:23.602
[mdnsd(28485)]CRS-5602:mDNS service stopping by request.
2011-07-02 18:17:36.217
[gpnpd(28732)]CRS-2328:GPNPD started on node rac2.
2011-07-02 18:17:43.673
[cssd(28794)]CRS-1713:CSSD daemon is started in clustered mode
2011-07-02 18:17:49.826
[cssd(28794)]CRS-1707:Lease acquisition for node rac2 number 2 completed
2011-07-02 18:17:49.865
[cssd(28794)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
2011-07-02 18:18:03.049
[cssd(28794)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2011-07-02 18:18:06.160
[ctssd(28861)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
2011-07-02 18:18:06.220
[ctssd(28861)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
2011-07-02 18:18:06.238
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 18:18:06.239
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 18:18:06.794
[ctssd(28861)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
[client(28891)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(28895)]CRS-10001:ACFS-9322: done.
2011-07-02 18:18:33.465
[crsd(29020)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:33.575
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:35.757
[crsd(29051)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:36.129
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:38.596
[crsd(29066)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:39.146
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:41.058
[crsd(29085)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:41.435
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:44.255
[crsd(29101)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:45.165
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:47.013
[crsd(29121)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:47.409
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:50.071
[crsd(29136)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:50.118
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:51.843
[crsd(29156)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:52.373
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:54.361
[crsd(29171)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:54.772
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:56.620
[crsd(29202)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:57.104
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:58.997
[crsd(29218)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
2011-07-02 18:18:59.301
[ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
2011-07-02 18:18:59.302
[ohasd(27261)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2011-07-02 18:49:58.070
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 18:49:58.070
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 19:21:33.362
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 19:21:33.362
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 19:52:05.271
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 19:52:05.271
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 20:22:53.696
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 20:22:53.696
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 20:53:43.949
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 20:53:43.949
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 21:24:32.990
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 21:24:32.990
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 21:55:21.907
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 21:55:21.908
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 22:26:45.752
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 22:26:45.752
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 22:57:54.682
[ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
2011-07-02 22:57:54.683
[ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2011-07-02 23:07:28.603
[cssd(28794)]CRS-1612:Network communication with node rac1 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.020 seconds
2011-07-02 23:07:35.621
[cssd(28794)]CRS-1611:Network communication with node rac1 (1) missing for 75% of timeout interval.  Removal of this node from cluster in 7.010 seconds
2011-07-02 23:07:39.629
[cssd(28794)]CRS-1610:Network communication with node rac1 (1) missing for 90% of timeout interval.  Removal of this node from cluster in 3.000 seconds
2011-07-02 23:07:42.641
[cssd(28794)]CRS-1632:Node rac1 is being removed from the cluster in cluster incarnation 205080558
2011-07-02 23:07:44.751
[cssd(28794)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
2011-07-02 23:07:45.326
[ctssd(28861)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
2011-07-04 19:46:26.008
[ohasd(27261)]CRS-8011:reboot advisory message from host: rac1, component: mo155738, with time stamp: L-2011-07-04-19:44:43.318
[ohasd(27261)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
*[root@rac2 rac2]#* This log file start with complaint that OLR is not accessible. Here is what I see (rca2):
-rw------- 1 root oinstall 272756736 Jul  2 18:18 /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olrAnd I guess rest of the problems start with this.

Similar Messages

  • RAC 11gR2 cluster installation: root.sh failed on the 1st node

    Hi,
    Does anybody know why is possible when I run the root.sh on the 1st node, during the Oracle 11gR2 RAC installation (cluster installation) to get the following error?
    The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /oracle/grid
    Enter the full pathname of the local bin directory: [usr/local/bin]:
    The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
    Copying dbhome to /usr/local/bin ...
    The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
    Copying oraenv to /usr/local/bin ...
    The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
    Copying coraenv to /usr/local/bin ...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-06-29 14:17:43: Parsing the host name
    2010-06-29 14:17:43: Checking for super user privileges
    2010-06-29 14:17:43: User has super user privileges
    Using configuration parameter file: /oracle/grid/crs/install/crsconfig_params
    Creating trace directory
    User oracle has the required capabilities to run CSSD in realtime mode
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'system'..
    Operation successful.
    root wallet
    root wallet cert
    root cert export
    peer wallet
    profile reader wallet
    pa wallet
    peer wallet keys
    pa wallet keys
    peer cert request
    pa cert request
    peer cert
    pa cert
    peer root cert TP
    profile reader root cert TP
    pa root cert TP
    peer pa cert TP
    pa peer cert TP
    profile reader pa cert TP
    profile reader peer cert TP
    peer user cert
    pa user cert
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-2672: Attempting to start 'ora.gipcd' on 'trz1test_rac'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'trz1test_rac'
    CRS-2676: Start of 'ora.gipcd' on 'trz1test_rac' succeeded
    CRS-2676: Start of 'ora.mdnsd' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'trz1test_rac'
    CRS-2676: Start of 'ora.gpnpd' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'trz1test_rac'
    CRS-2676: Start of 'ora.cssdmonitor' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'trz1test_rac'
    CRS-2672: Attempting to start 'ora.diskmon' on 'trz1test_rac'
    CRS-2676: Start of 'ora.diskmon' on 'trz1test_rac' succeeded
    CRS-2676: Start of 'ora.cssd' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'trz1test_rac'
    CRS-2676: Start of 'ora.ctssd' on 'trz1test_rac' succeeded
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'system'..
    Operation successful.
    CRS-2672: Attempting to start 'ora.crsd' on 'trz1test_rac'
    CRS-2676: Start of 'ora.crsd' on 'trz1test_rac' succeeded
    Now formatting voting disk: /data_gpfs/oracle/crs/vdsk.
    CRS-4603: Successful addition of voting disk /data_gpfs/oracle/crs/vdsk.
    ## STATE File Universal Id File Name Disk group
    1. ONLINE 653624f2aa1f4f83bf774e8052889a32 (/data_gpfs/oracle/crs/vdsk) []
    Located 1 voting disk(s).
    CRS-2673: Attempting to stop 'ora.crsd' on 'trz1test_rac'
    CRS-2677: Stop of 'ora.crsd' on 'trz1test_rac' succeeded
    CRS-2673: Attempting to stop 'ora.ctssd' on 'trz1test_rac'
    CRS-2677: Stop of 'ora.ctssd' on 'trz1test_rac' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'trz1test_rac'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'trz1test_rac' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'trz1test_rac'
    CRS-2677: Stop of 'ora.cssd' on 'trz1test_rac' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'trz1test_rac'
    CRS-2677: Stop of 'ora.gpnpd' on 'trz1test_rac' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'trz1test_rac'
    CRS-2677: Stop of 'ora.gipcd' on 'trz1test_rac' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'trz1test_rac'
    CRS-2677: Stop of 'ora.mdnsd' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.mdnsd' on 'trz1test_rac'
    CRS-2676: Start of 'ora.mdnsd' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'trz1test_rac'
    CRS-2676: Start of 'ora.gipcd' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'trz1test_rac'
    CRS-2676: Start of 'ora.gpnpd' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'trz1test_rac'
    CRS-2676: Start of 'ora.cssdmonitor' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'trz1test_rac'
    CRS-2672: Attempting to start 'ora.diskmon' on 'trz1test_rac'
    CRS-2676: Start of 'ora.diskmon' on 'trz1test_rac' succeeded
    CRS-2676: Start of 'ora.cssd' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'trz1test_rac'
    CRS-2676: Start of 'ora.ctssd' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.crsd' on 'trz1test_rac'
    CRS-2676: Start of 'ora.crsd' on 'trz1test_rac' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 'trz1test_rac'
    CRS-2676: Start of 'ora.evmd' on 'trz1test_rac' succeeded
    */oracle/grid/bin/srvctl start nodeapps -n trz1test_rac ... failed*
    Configure Oracle Grid Infrastructure for a Cluster ... failed
    This is because ora.eONS daemon is not starting. There is a Metalink note that we MIGHT start this daemon manually ... but this is not working.
    *./srvctl status nodeapps -n trz1test_rac*
    -n <node_name> option has been deprecated.
    VIP trz1test_rac_vip is enabled
    VIP trz1test_rac_vip is running on node: trz1test_rac
    Network is enabled
    Network is running on node: trz1test_rac
    GSD is disabled
    GSD is not running on node: trz1test_rac
    ONS is enabled
    ONS daemon is running on node: trz1test_rac
    eONS is enabled
    eONS daemon is not running on node: trz1test_rac

    I run my clusterware/DB on AIX 5.3
    When I run runcluvfy.sh here are the things which are not passing:
    Check: Node connectivity of subnet "192.168.1.0"
    Source Destination Connected?
    trz2test_rac:en5 trz2test_rac:en5 yes
    trz2test_rac:en5 trz1test_rac:en5 yes
    trz2test_rac:en5 trz1test_rac:en5 yes
    trz2test_rac:en5 trz1test_rac:en5 yes
    trz2test_rac:en5 trz1test_rac:en5 yes
    trz1test_rac:en5 trz1test_rac:en5 yes
    Result: Node connectivity passed for subnet "192.168.1.0" with node(s) trz2test_rac,trz1test_rac
    Check: TCP connectivity of subnet "192.168.1.0"
    Source Destination Connected?
    trz1test_rac:192.168.1.140 trz2test_rac:192.168.1.142 failed
    trz1test_rac:192.168.1.140 trz2test_rac:192.168.1.142 failed
    Result: TCP connectivity check failed for subnet "192.168.1.0"
    NTP daemon slewing option check failed on some nodes
    PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing option "-x"
    Result: Clock synchronization check using Network Time Protocol(NTP) failed
    NTP mustn't be a problem I guess as the date are identical on the 2 nodes.
    I have no idea how to fix the TCP connectivity issue with the subnet "192.168.1.0". Some posts wrote that could be a firewall issue. Are there any other causes ?
    Thanks to all,
    Paul

  • Command return code of 255 (65280) during Grid Infrastructure Installation

    I am trying to create a 2-node RAC 11gR2. I am using OEL 5 Update 4. I am using NFS as shared storage. The time on both node is same. Both nodes have 4GB RAM each. Following is my IP settings from /etc/hosts on both nodes:
    [oracle@vis ~]$ cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1 localhost.localdomain localhost
    # Public
    10.8.5.191 vis.awc.com vis
    10.8.5.192 vsstest.awc.com vsstest
    #Private
    192.168.0.101 vis-priv.awc.com vis-priv
    192.168.0.102 vsstest-priv.awc.com vsstest-priv
    #Virtual
    10.8.5.166 vis-vip.awc.com vis-vip
    10.8.5.170 vsstest-vip.awc.com vsstest-vip
    # SCAN
    10.8.5.182 rac-scan.awc.com rac-scan
    #NAS
    192.168.0.101 nas1.awc.com nas1
    I am getting error during the Grid Infrastructure installation, when I run the root.sh script through root. Following is the logfile showing the error (/u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_vis.log):
    2010-02-02 11:58:05: The configuration parameter file /u01/app/11.2.0/grid/crs/install/crsconfig_params is valid
    2010-02-02 11:58:05: Checking for super user privileges
    2010-02-02 11:58:05: User has super user privileges
    2010-02-02 11:58:05: ### Printing the configuration values from files:
    2010-02-02 11:58:05: /u01/app/11.2.0/grid/crs/install/crsconfig_params
    2010-02-02 11:58:05: /u01/app/11.2.0/grid/crs/install/s_crsconfig_defs
    2010-02-02 11:58:05: ASM_DISCOVERY_STRING=
    2010-02-02 11:58:05: ASM_DISKS=
    2010-02-02 11:58:05: ASM_DISK_GROUP=
    2010-02-02 11:58:05: ASM_REDUNDANCY=
    2010-02-02 11:58:05: ASM_SPFILE=
    2010-02-02 11:58:05: ASM_UPGRADE=false
    2010-02-02 11:58:05: CLSCFG_MISSCOUNT=
    2010-02-02 11:58:05: CLUSTER_GUID=
    2010-02-02 11:58:05: CLUSTER_NAME=rac-scan
    2010-02-02 11:58:05: CRS_NODEVIPS='vis-vip/255.248.0.0/eth0,vsstest-vip/255.248.0.0/eth0'
    2010-02-02 11:58:05: CRS_STORAGE_OPTION=2
    2010-02-02 11:58:05: CSS_LEASEDURATION=400
    2010-02-02 11:58:05: DIRPREFIX=
    2010-02-02 11:58:05: DISABLE_OPROCD=0
    2010-02-02 11:58:05: EMBASEJAR_NAME=oemlt.jar
    2010-02-02 11:58:05: EWTJAR_NAME=ewt3.jar
    2010-02-02 11:58:05: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
    2010-02-02 11:58:05: GNS_ADDR_LIST=
    2010-02-02 11:58:05: GNS_ALLOW_NET_LIST=
    2010-02-02 11:58:05: GNS_CONF=false
    2010-02-02 11:58:05: GNS_DENY_ITF_LIST=
    2010-02-02 11:58:05: GNS_DENY_NET_LIST=
    2010-02-02 11:58:05: GNS_DOMAIN_LIST=
    2010-02-02 11:58:05: GPNPCONFIGDIR=/u01/app/11.2.0/grid
    2010-02-02 11:58:05: GPNPGCONFIGDIR=/u01/app/11.2.0/grid
    2010-02-02 11:58:05: GPNP_PA=
    2010-02-02 11:58:05: HELPJAR_NAME=help4.jar
    2010-02-02 11:58:05: HOST_NAME_LIST=vis,vsstest
    2010-02-02 11:58:05: ID=/etc/init.d
    2010-02-02 11:58:05: INIT=/sbin/init
    2010-02-02 11:58:05: IT=/etc/inittab
    2010-02-02 11:58:05: JEWTJAR_NAME=jewt4.jar
    2010-02-02 11:58:05: JLIBDIR=/u01/app/11.2.0/grid/jlib
    2010-02-02 11:58:05: JREDIR=/u01/app/11.2.0/grid/jdk/jre/
    2010-02-02 11:58:06: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
    2010-02-02 11:58:06: MSGFILE=/var/adm/messages
    2010-02-02 11:58:06: NETCFGJAR_NAME=netcfg.jar
    2010-02-02 11:58:06: NETWORKS="eth0"/10.8.0.0:public,"eth1"/192.168.0.0:cluster_interconnect
    2010-02-02 11:58:06: NEW_HOST_NAME_LIST=
    2010-02-02 11:58:06: NEW_NODEVIPS='vis-vip/255.248.0.0/eth0,vsstest-vip/255.248.0.0/eth0'
    2010-02-02 11:58:06: NEW_NODE_NAME_LIST=
    2010-02-02 11:58:06: NEW_PRIVATE_NAME_LIST=
    2010-02-02 11:58:06: NODELIST=vis,vsstest
    2010-02-02 11:58:06: NODE_NAME_LIST=vis,vsstest
    2010-02-02 11:58:06: OCFS_CONFIG=
    2010-02-02 11:58:06: OCRCONFIG=/etc/oracle/ocr.loc
    2010-02-02 11:58:06: OCRCONFIGDIR=/etc/oracle
    2010-02-02 11:58:06: OCRID=
    2010-02-02 11:58:06: OCRLOC=ocr.loc
    2010-02-02 11:58:06: OCR_LOCATIONS=/u01/shared_config/ocr_configuration
    2010-02-02 11:58:06: OLASTGASPDIR=/etc/oracle/lastgasp
    2010-02-02 11:58:06: OLRCONFIG=/etc/oracle/olr.loc
    2010-02-02 11:58:06: OLRCONFIGDIR=/etc/oracle
    2010-02-02 11:58:06: OLRLOC=olr.loc
    2010-02-02 11:58:06: OPROCDCHECKDIR=/etc/oracle/oprocd/check
    2010-02-02 11:58:06: OPROCDDIR=/etc/oracle/oprocd
    2010-02-02 11:58:06: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
    2010-02-02 11:58:06: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
    2010-02-02 11:58:06: ORACLE_BASE=/u01/app/oracle
    2010-02-02 11:58:06: ORACLE_HOME=/u01/app/11.2.0/grid
    2010-02-02 11:58:06: ORACLE_OWNER=oracle
    2010-02-02 11:58:06: ORA_ASM_GROUP=dba
    2010-02-02 11:58:06: ORA_DBA_GROUP=oinstall
    2010-02-02 11:58:06: PRIVATE_NAME_LIST=
    2010-02-02 11:58:06: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
    2010-02-02 11:58:06: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
    2010-02-02 11:58:06: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
    2010-02-02 11:58:06: RC_KILL=K19
    2010-02-02 11:58:06: RC_KILL_OLD=K96
    2010-02-02 11:58:06: RC_START=S96
    2010-02-02 11:58:06: SCAN_NAME=rac-scan.awc.com
    2010-02-02 11:58:06: SCAN_PORT=1521
    2010-02-02 11:58:06: SCRBASE=/etc/oracle/scls_scr
    2010-02-02 11:58:06: SHAREJAR_NAME=share.jar
    2010-02-02 11:58:06: SILENT=false
    2010-02-02 11:58:06: SO_EXT=so
    2010-02-02 11:58:06: SRVCFGLOC=srvConfig.loc
    2010-02-02 11:58:06: SRVCONFIG=/var/opt/oracle/srvConfig.loc
    2010-02-02 11:58:06: SRVCONFIGDIR=/var/opt/oracle
    2010-02-02 11:58:06: VNDR_CLUSTER=false
    2010-02-02 11:58:06: VOTING_DISKS=/u01/shared_config/voting_disk
    2010-02-02 11:58:06: ### Printing other configuration values ###
    2010-02-02 11:58:06: CLSCFG_EXTRA_PARMS=
    2010-02-02 11:58:06: CRSDelete=0
    2010-02-02 11:58:06: CRSPatch=0
    2010-02-02 11:58:06: DEBUG=
    2010-02-02 11:58:06: DOWNGRADE=
    2010-02-02 11:58:06: HAS_GROUP=oinstall
    2010-02-02 11:58:06: HAS_USER=root
    2010-02-02 11:58:06: HOST=vis
    2010-02-02 11:58:07: IS_SIHA=0
    2010-02-02 11:58:07: OLR_DIRECTORY=/u01/app/11.2.0/grid/cdata
    2010-02-02 11:58:07: OLR_LOCATION=/u01/app/11.2.0/grid/cdata/vis.olr
    2010-02-02 11:58:07: ORA_CRS_HOME=/u01/app/11.2.0/grid
    2010-02-02 11:58:07: SUPERUSER=root
    2010-02-02 11:58:07: UPGRADE=
    2010-02-02 11:58:07: VF_DISCOVERY_STRING=/u01/shared_config/voting_disk
    2010-02-02 11:58:07: addfile=/u01/app/11.2.0/grid/crs/install/crsconfig_addparams
    2010-02-02 11:58:07: crscfg_trace=1
    2010-02-02 11:58:07: crscfg_trace_file=/u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_vis.log
    2010-02-02 11:58:07: hosts=
    2010-02-02 11:58:07: oldcrshome=
    2010-02-02 11:58:07: oldcrsver=
    2010-02-02 11:58:07: osdfile=/u01/app/11.2.0/grid/crs/install/s_crsconfig_defs
    2010-02-02 11:58:07: parameters_valid=1
    2010-02-02 11:58:07: paramfile=/u01/app/11.2.0/grid/crs/install/crsconfig_params
    2010-02-02 11:58:07: platform_family=unix
    2010-02-02 11:58:07: srvctl_trc_suff=0
    2010-02-02 11:58:07: unlock_crshome=
    2010-02-02 11:58:07: user_is_superuser=1
    2010-02-02 11:58:07: ### Printing of configuration values complete ###
    2010-02-02 11:58:07: Oracle CRS stack is not configured yet
    2010-02-02 11:58:07: CRS is not yet configured. Hence, will proceed to configure CRS
    2010-02-02 11:58:07: Cluster-wide one-time actions... Done!
    2010-02-02 11:58:17: set owner/group of OCR path
    2010-02-02 11:58:17: Oracle CRS home = /u01/app/11.2.0/grid
    2010-02-02 11:58:17: Host name = vis
    2010-02-02 11:58:17: CRS user = oracle
    2010-02-02 11:58:17: Oracle CRS home = /u01/app/11.2.0/grid
    2010-02-02 11:58:17: GPnP host = vis
    2010-02-02 11:58:17: Oracle GPnP home = /u01/app/11.2.0/grid/gpnp
    2010-02-02 11:58:17: Oracle GPnP local home = /u01/app/11.2.0/grid/gpnp/vis
    2010-02-02 11:58:17: GPnP directories verified.
    2010-02-02 11:58:17: Checking to see if Oracle CRS stack is already configured
    2010-02-02 11:58:17: Oracle CRS stack is not configured yet
    2010-02-02 11:58:17: ---Checking local gpnp setup...
    2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" does not exist
    2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cwallet.sso" does not exist
    2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/vis/wallets/prdr/cwallet.sso" does not exist
    2010-02-02 11:58:18: chk gpnphome /u01/app/11.2.0/grid/gpnp/vis: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
    2010-02-02 11:58:18: chk gpnphome /u01/app/11.2.0/grid/gpnp/vis: INVALID (bad profile/wallet)
    2010-02-02 11:58:18: ---Checking cluster-wide gpnp setup...
    2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/profiles/peer/profile.xml" does not exist
    2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/wallets/peer/cwallet.sso" does not exist
    2010-02-02 11:58:18: The setup file "/u01/app/11.2.0/grid/gpnp/wallets/prdr/cwallet.sso" does not exist
    2010-02-02 11:58:18: chk gpnphome /u01/app/11.2.0/grid/gpnp: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
    2010-02-02 11:58:18: chk gpnphome /u01/app/11.2.0/grid/gpnp: INVALID (bad profile/wallet)
    2010-02-02 11:58:18: gpnp setup checked: local valid? 0 cluster-wide valid? 0
    2010-02-02 11:58:18: gpnp setup: NONE
    2010-02-02 11:58:18: GPNP configuration required
    2010-02-02 11:58:18: Validating for SI-CSS configuration
    2010-02-02 11:58:18: Retrieving OCR main disk location
    2010-02-02 11:58:18: Opening file OCRCONFIG
    2010-02-02 11:58:18: Value () is set for key=ocrconfig_loc
    2010-02-02 11:58:18: Unable to retrieve ocr disk info
    2010-02-02 11:58:18: Checking to see if any 9i GSD is up
    2010-02-02 11:58:18: libskgxnBase_lib = /etc/ORCLcluster/oracm/lib/libskgxn2.so
    2010-02-02 11:58:18: libskgxn_lib = /opt/ORCLcluster/lib/libskgxn2.so
    2010-02-02 11:58:18: SKGXN library file does not exists
    2010-02-02 11:58:18: OLR location = /u01/app/11.2.0/grid/cdata/vis.olr
    2010-02-02 11:58:18: Oracle CRS Home = /u01/app/11.2.0/grid
    2010-02-02 11:58:18: Validating /etc/oracle/olr.loc file for OLR location /u01/app/11.2.0/grid/cdata/vis.olr
    2010-02-02 11:58:18: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig
    2010-02-02 11:58:18: Oracle CRS home = /u01/app/11.2.0/grid
    2010-02-02 11:58:18: Oracle cluster name = rac-scan
    2010-02-02 11:58:18: OCR locations = /u01/shared_config/ocr_configuration
    2010-02-02 11:58:18: Validating OCR
    2010-02-02 11:58:18: Retrieving OCR location used by previous installations
    2010-02-02 11:58:18: Opening file OCRCONFIG
    2010-02-02 11:58:18: Value () is set for key=ocrconfig_loc
    2010-02-02 11:58:18: Opening file OCRCONFIG
    2010-02-02 11:58:18: Value () is set for key=ocrmirrorconfig_loc
    2010-02-02 11:58:18: Opening file OCRCONFIG
    2010-02-02 11:58:18: Value () is set for key=ocrconfig_loc3
    2010-02-02 11:58:18: Opening file OCRCONFIG
    2010-02-02 11:58:18: Value () is set for key=ocrconfig_loc4
    2010-02-02 11:58:18: Opening file OCRCONFIG
    2010-02-02 11:58:18: Value () is set for key=ocrconfig_loc5
    2010-02-02 11:58:18: Checking if OCR sync file exists
    2010-02-02 11:58:18: No need to sync OCR file
    2010-02-02 11:58:18: OCR_LOCATION=/u01/shared_config/ocr_configuration
    2010-02-02 11:58:18: OCR_MIRROR_LOCATION=
    2010-02-02 11:58:18: OCR_MIRROR_LOC3=
    2010-02-02 11:58:18: OCR_MIRROR_LOC4=
    2010-02-02 11:58:19: OCR_MIRROR_LOC5=
    2010-02-02 11:58:19: Current OCR location=
    2010-02-02 11:58:19: Current OCR mirror location=
    2010-02-02 11:58:19: Current OCR mirror loc3=
    2010-02-02 11:58:19: Current OCR mirror loc4=
    2010-02-02 11:58:19: Current OCR mirror loc5=
    2010-02-02 11:58:19: Verifying current OCR settings with user entered values
    2010-02-02 11:58:19: Setting OCR locations in /etc/oracle/ocr.loc
    2010-02-02 11:58:19: Validating OCR locations in /etc/oracle/ocr.loc
    2010-02-02 11:58:19: Checking for existence of /etc/oracle/ocr.loc
    2010-02-02 11:58:19: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig
    2010-02-02 11:58:19: Setting ocr location /u01/shared_config/ocr_configuration
    2010-02-02 11:58:19: Creating or upgrading Oracle Local Registry (OLR)
    2010-02-02 11:58:20: OLR successfully created or upgraded
    2010-02-02 11:58:20: /u01/app/11.2.0/grid/bin/clscfg -localadd
    2010-02-02 11:58:21: Keys created in the OLR successfully
    2010-02-02 11:58:21: GPnP setup state: none
    2010-02-02 11:58:21: Creating local GPnP setup for clustered node...
    2010-02-02 11:58:21: Oracle CRS home = /u01/app/11.2.0/grid
    2010-02-02 11:58:21: Oracle GPnP wallets home = /u01/app/11.2.0/grid/gpnp/vis/wallets
    2010-02-02 11:58:21: Checking if GPnP setup exists
    2010-02-02 11:58:21: /u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cwallet.sso wallet must be created
    2010-02-02 11:58:21: Removing old wallets/certificates, if any
    2010-02-02 11:58:21: Creating GPnP Root Wallet...
    2010-02-02 11:58:21: /u01/app/11.2.0/grid/bin/orapki wallet create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/root" -pwd gpnp_wallet1 -nologo
    2010-02-02 11:58:22: Creating GPnP Root Certificate...
    2010-02-02 11:58:22: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/root" -pwd gpnp_wallet1 -self_signed -dn "CN=GPnP_root" -keysize 1024 -validity 9999 -nologo
    2010-02-02 11:58:24: Exporting GPnP Root Certificate...
    2010-02-02 11:58:24: /u01/app/11.2.0/grid/bin/orapki wallet export -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/root" -pwd gpnp_wallet1 -dn "CN=GPnP_root" -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/root/b64certificate.txt" -nologo
    2010-02-02 11:58:25: Creating GPnP Peer Wallet...
    2010-02-02 11:58:25: /u01/app/11.2.0/grid/bin/orapki wallet create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -auto_login -nologo
    2010-02-02 11:58:25: Creating GPnP Profile Reader Wallet...
    2010-02-02 11:58:25: /u01/app/11.2.0/grid/bin/orapki wallet create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/prdr" -pwd gpnp_wallet1 -auto_login -nologo
    2010-02-02 11:58:26: Creating GPnP PA Wallet...
    2010-02-02 11:58:26: /u01/app/11.2.0/grid/bin/orapki wallet create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -auto_login -nologo
    2010-02-02 11:58:27: Adding private key to GPnP Peer Wallet...
    2010-02-02 11:58:27: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -dn "CN=GPnP_peer" -keysize 1024 -nologo
    2010-02-02 11:58:28: Adding private key to GPnP PA Wallet...
    2010-02-02 11:58:28: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -dn "CN=GPnP_pa" -keysize 1024 -nologo
    2010-02-02 11:58:29: Creating certificate request for GPnP Peer Wallet...
    2010-02-02 11:58:29: /u01/app/11.2.0/grid/bin/orapki wallet export -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -dn "CN=GPnP_peer" -request "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/certreq.txt" -nologo
    2010-02-02 11:58:30: Creating certificate request for GPnP PA Wallet...
    2010-02-02 11:58:30: /u01/app/11.2.0/grid/bin/orapki wallet export -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -dn "CN=GPnP_pa" -request "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/certreq.txt" -nologo
    2010-02-02 11:58:31: Creating certificate for GPnP Peer Wallet...
    2010-02-02 11:58:31: /u01/app/11.2.0/grid/bin/orapki cert create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/root" -pwd gpnp_wallet1 -request "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/certreq.txt" -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cert.txt" -validity 9999 -nologo
    2010-02-02 11:58:32: Creating certificate for GPnP PA Wallet...
    2010-02-02 11:58:32: /u01/app/11.2.0/grid/bin/orapki cert create -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/root" -pwd gpnp_wallet1 -request "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/certreq.txt" -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/cert.txt" -validity 9999 -nologo
    2010-02-02 11:58:33: Adding Root Certificate TP to GPnP Peer Wallet...
    2010-02-02 11:58:33: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/root/b64certificate.txt" -nologo
    2010-02-02 11:58:34: Adding Root Certificate TP to GPnP Profile Reader Wallet...
    2010-02-02 11:58:34: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/prdr" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/root/b64certificate.txt" -nologo
    2010-02-02 11:58:34: Adding Root Certificate TP to GPnP PA Wallet...
    2010-02-02 11:58:34: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/root/b64certificate.txt" -nologo
    2010-02-02 11:58:35: Adding PA Certificate as a TP into a GPnP Peer Wallet...
    2010-02-02 11:58:35: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/cert.txt" -nologo
    2010-02-02 11:58:36: Adding peer Certificate as a TP into a GPnP PA Wallet...
    2010-02-02 11:58:36: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cert.txt" -nologo
    2010-02-02 11:58:37: Adding PA Certificate as a TP into a GPnP Profile Reader Wallet...
    2010-02-02 11:58:37: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/prdr" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/cert.txt" -nologo
    2010-02-02 11:58:38: Adding peer Certificate as a TP into a GPnP Profile Reader Wallet...
    2010-02-02 11:58:38: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/prdr" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cert.txt" -nologo
    2010-02-02 11:58:39: Adding PA Certificate as a TP into a GPnP Peer Wallet...
    2010-02-02 11:58:39: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -pwd gpnp_wallet1 -user_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/peer/cert.txt" -nologo
    2010-02-02 11:58:40: Adding peer Certificate as a TP into a GPnP PA Wallet...
    2010-02-02 11:58:40: /u01/app/11.2.0/grid/bin/orapki wallet add -wallet "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa" -pwd gpnp_wallet1 -user_cert -cert "/u01/app/11.2.0/grid/gpnp/vis/wallets/pa/cert.txt" -nologo
    2010-02-02 11:58:41: GPnP Wallets ownership/permissions successfully set.
    2010-02-02 11:58:41: GPnP Wallets successfully created.
    2010-02-02 11:58:41: <--- GPnP wallets successfully created
    2010-02-02 11:58:41: Creating GPnP peer profile --->
    2010-02-02 11:58:41: Oracle CRS home = /u01/app/11.2.0/grid
    2010-02-02 11:58:41: Oracle GPnP profiles home = /u01/app/11.2.0/grid/gpnp/vis/profiles
    2010-02-02 11:58:41: Oracle GPnP profiles parameters:
    2010-02-02 11:58:41: paloc==
    2010-02-02 11:58:41: cname=rac-scan=
    2010-02-02 11:58:41: cssdisco=/u01/shared_config/voting_disk=
    2010-02-02 11:58:41: cssld=400=
    2010-02-02 11:58:41: asmdisco==
    2010-02-02 11:58:41: asmspf==
    2010-02-02 11:58:41: netlst="eth0"/10.8.0.0:public,"eth1"/192.168.0.0:cluster_interconnect=
    2010-02-02 11:58:41: ocrid==
    2010-02-02 11:58:41: clusterguid==
    2010-02-02 11:58:41: Checking if GPnP setup exists
    2010-02-02 11:58:41: /u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml profile must be created
    2010-02-02 11:58:41: OCRID is not available, hence not set in GPnP Profile
    2010-02-02 11:58:42: ClusterGUID is not available, hence not set in GPnP Profile
    2010-02-02 11:58:42: gpnptool: run /u01/app/11.2.0/grid/bin/gpnptool create -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -prf -prf_sq=1 -prf_cn=rac-scan -prf_pa="" -hnet=gen -gen:hnet_nm="*" -gen:net=net1 -net1:net_ip="10.8.0.0" -net1:net_ada="eth0" -net1:net_use="public" -gen:net=net2 -net2:net_ip="192.168.0.0" -net2:net_ada="eth1" -net2:net_use="cluster_interconnect" -css=css -css:css_dis="/u01/shared_config/voting_disk" -css:css_ld=400 -asm=asm -asm:asm_dis="++no-value-at-profile-creation--never-updated-through-ASM++" -asm:asm_spf=""
    2010-02-02 11:58:42: Running as user oracle: /u01/app/11.2.0/grid/bin/gpnptool create -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -prf -prf_sq=1 -prf_cn=rac-scan -prf_pa="" -hnet=gen -gen:hnet_nm="*" -gen:net=net1 -net1:net_ip="10.8.0.0" -net1:net_ada="eth0" -net1:net_use="public" -gen:net=net2 -net2:net_ip="192.168.0.0" -net2:net_ada="eth1" -net2:net_use="cluster_interconnect" -css=css -css:css_dis="/u01/shared_config/voting_disk" -css:css_ld=400 -asm=asm -asm:asm_dis="++no-value-at-profile-creation--never-updated-through-ASM++" -asm:asm_spf=""
    2010-02-02 11:58:42: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/11.2.0/grid/bin/gpnptool create -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -prf -prf_sq=1 -prf_cn=rac-scan -prf_pa="" -hnet=gen -gen:hnet_nm="*" -gen:net=net1 -net1:net_ip="10.8.0.0" -net1:net_ada="eth0" -net1:net_use="public" -gen:net=net2 -net2:net_ip="192.168.0.0" -net2:net_ada="eth1" -net2:net_use="cluster_interconnect" -css=css -css:css_dis="/u01/shared_config/voting_disk" -css:css_ld=400 -asm=asm -asm:asm_dis="++no-value-at-profile-creation--never-updated-through-ASM++" -asm:asm_spf="" '
    2010-02-02 11:58:42: Removing file /tmp/fileXUKYkK
    2010-02-02 11:58:42: Successfully removed file: /tmp/fileXUKYkK
    2010-02-02 11:58:42: /bin/su successfully executed
    2010-02-02 11:58:42: gpnptool: rc=0
    2010-02-02 11:58:42: gpnptool output:
    Resulting profile written to "/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml".
    Success.
    2010-02-02 11:58:42: gpnptool: run /u01/app/11.2.0/grid/bin/gpnptool sign -p="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -w="file:/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -rmws
    2010-02-02 11:58:42: Running as user oracle: /u01/app/11.2.0/grid/bin/gpnptool sign -p="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -w="file:/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -rmws
    2010-02-02 11:58:42: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/11.2.0/grid/bin/gpnptool sign -p="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -o="/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml" -ovr -w="file:/u01/app/11.2.0/grid/gpnp/vis/wallets/peer" -rmws '
    2010-02-02 11:58:43: Removing file /tmp/file4E7Vnj
    2010-02-02 11:58:43: Successfully removed file: /tmp/file4E7Vnj
    2010-02-02 11:58:43: /bin/su successfully executed
    2010-02-02 11:58:43: gpnptool: rc=0
    2010-02-02 11:58:43: gpnptool output:
    Resulting profile written to "/u01/app/11.2.0/grid/gpnp/vis/profiles/peer/profile.xml".
    Success.
    2010-02-02 11:58:43: GPnP peer profile create successfully completed.
    2010-02-02 11:58:43: <--- GPnP peer profile successfully created
    2010-02-02 11:58:43: GPnP local setup successfully created
    2010-02-02 11:58:43: Registering ohasd
    2010-02-02 11:58:43: init file = /u01/app/11.2.0/grid/crs/init/init.ohasd
    2010-02-02 11:58:43: Copying file /u01/app/11.2.0/grid/crs/init/init.ohasd to /etc/init.d directory
    2010-02-02 11:58:43: Setting init.ohasd permission in /etc/init.d directory
    2010-02-02 11:58:43: init file = /u01/app/11.2.0/grid/crs/init/ohasd
    2010-02-02 11:58:43: Copying file /u01/app/11.2.0/grid/crs/init/ohasd to /etc/init.d directory
    2010-02-02 11:58:43: Setting ohasd permission in /etc/init.d directory
    2010-02-02 11:58:43: Removing "/etc/rc.d/rc3.d/S96ohasd"
    2010-02-02 11:58:43: Removing file /etc/rc.d/rc3.d/S96ohasd
    2010-02-02 11:58:43: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd
    2010-02-02 11:58:43: Failed to remove file:
    2010-02-02 11:58:43: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
    2010-02-02 11:58:43: Removing "/etc/rc.d/rc5.d/S96ohasd"
    2010-02-02 11:58:43: Removing file /etc/rc.d/rc5.d/S96ohasd
    2010-02-02 11:58:43: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd
    2010-02-02 11:58:43: Failed to remove file:
    2010-02-02 11:58:43: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
    2010-02-02 11:58:43: Removing "/etc/rc.d/rc0.d/K19ohasd"
    2010-02-02 11:58:43: Removing file /etc/rc.d/rc0.d/K19ohasd
    2010-02-02 11:58:43: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd
    2010-02-02 11:58:43: Failed to remove file:
    2010-02-02 11:58:43: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd
    2010-02-02 11:58:43: Removing "/etc/rc.d/rc1.d/K19ohasd"
    2010-02-02 11:58:43: Removing file /etc/rc.d/rc1.d/K19ohasd
    2010-02-02 11:58:43: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd
    2010-02-02 11:58:44: Failed to remove file:
    2010-02-02 11:58:44: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd
    2010-02-02 11:58:44: Removing "/etc/rc.d/rc2.d/K19ohasd"
    2010-02-02 11:58:44: Removing file /etc/rc.d/rc2.d/K19ohasd
    2010-02-02 11:58:44: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd
    2010-02-02 11:58:44: Failed to remove file:
    2010-02-02 11:58:44: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd
    2010-02-02 11:58:44: Removing "/etc/rc.d/rc4.d/K19ohasd"
    2010-02-02 11:58:44: Removing file /etc/rc.d/rc4.d/K19ohasd
    2010-02-02 11:58:44: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd
    2010-02-02 11:58:44: Failed to remove file:
    2010-02-02 11:58:44: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd
    2010-02-02 11:58:44: Removing "/etc/rc.d/rc6.d/K19ohasd"
    2010-02-02 11:58:44: Removing file /etc/rc.d/rc6.d/K19ohasd
    2010-02-02 11:58:44: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
    2010-02-02 11:58:44: Failed to remove file:
    2010-02-02 11:58:44: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
    2010-02-02 11:58:44: The file ohasd has been successfully linked to the RC directories
    2010-02-02 11:58:44: Starting ohasd
    2010-02-02 11:58:44: itab entries=
    2010-02-02 11:58:49: Created backup /etc/inittab.no_crs
    2010-02-02 11:58:49: Appending to /etc/inittab.tmp:
    2010-02-02 11:58:50: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
    2010-02-02 11:58:50: Done updating /etc/inittab.tmp
    2010-02-02 11:58:50: Saved /etc/inittab.crs
    2010-02-02 11:58:50: Installed new /etc/inittab
    2010-02-02 11:58:55: ohasd is starting
    2010-02-02 11:58:55: Checking ohasd
    2010-02-02 11:58:55: ohasd started successfully
    2010-02-02 11:58:55: Creating CRS resources and dependencies
    2010-02-02 11:58:55: Configuring HASD
    2010-02-02 11:58:55: Registering type ora.daemon.type
    2010-02-02 11:58:59: Registering type ora.mdns.type
    2010-02-02 11:59:00: Registering type ora.gpnp.type
    2010-02-02 11:59:01: Registering type ora.gipc.type
    2010-02-02 11:59:02: Registering type ora.cssd.type
    2010-02-02 11:59:04: Registering type ora.cssdmonitor.type
    2010-02-02 11:59:07: Registering type ora.crs.type
    2010-02-02 11:59:08: Registering type ora.evm.type
    2010-02-02 11:59:09: Registering type ora.ctss.type
    2010-02-02 11:59:10: Registering type ora.asm.type
    2010-02-02 11:59:13: Registering type ora.drivers.acfs.type
    2010-02-02 11:59:14: Registering type ora.diskmon.type
    2010-02-02 11:59:34: ADVM/ACFS is configured
    2010-02-02 11:59:35: Successfully created CRS resources for cluster daemon and ASM
    2010-02-02 11:59:35: Checking if initial configuration has been performed
    2010-02-02 11:59:35: Starting CSS in exclusive mode
    2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.gipcd' on 'vis'
    2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.mdnsd' on 'vis'
    2010-02-02 12:00:04: CRS-2676: Start of 'ora.gipcd' on 'vis' succeeded
    2010-02-02 12:00:04: CRS-2676: Start of 'ora.mdnsd' on 'vis' succeeded
    2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.gpnpd' on 'vis'
    2010-02-02 12:00:04: CRS-2676: Start of 'ora.gpnpd' on 'vis' succeeded
    2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'vis'
    2010-02-02 12:00:04: CRS-2676: Start of 'ora.cssdmonitor' on 'vis' succeeded
    2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.cssd' on 'vis'
    2010-02-02 12:00:04: CRS-2672: Attempting to start 'ora.diskmon' on 'vis'
    2010-02-02 12:00:04: CRS-2676: Start of 'ora.diskmon' on 'vis' succeeded
    2010-02-02 12:00:04: CRS-2676: Start of 'ora.cssd' on 'vis' succeeded
    2010-02-02 12:00:04: Querying for existing CSS voting disks
    2010-02-02 12:00:04: Performing initial configuration for cluster
    2010-02-02 12:00:51: Start of resource "ora.ctssd -init" Succeeded
    2010-02-02 12:00:51: Creating or upgrading OCR keys
    2010-02-02 12:00:51: Command return code of 255 (65280) from command: /u01/app/11.2.0/grid/bin/ocrconfig -upgrade oracle oinstall
    2010-02-02 12:00:51: Failed to create Oracle Cluster Registry configuration, rc 255
    2010-02-02 12:00:51: Exiting exclusive mode
    2010-02-02 12:00:51: Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
    2010-02-02 12:00:51: Stop of resource "ora.crsd -init" failed
    2010-02-02 12:00:51: Failed to stop CRSD
    2010-02-02 12:01:20: Initial cluster configuration failed. See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_vis.log for details

    Hi,
    I am trying to setup Oracle 11gR2 RAC setup on RHEL 5.4 64-bit machine, during grid installation i am also getting the same error.
    2010-07-12 03:12:42: Removing file /etc/rc.d/rc6.d/K19ohasd
    2010-07-12 03:12:42: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
    2010-07-12 03:12:42: Failed to remove file:
    2010-07-12 03:12:42: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
    2010-07-12 03:12:42: The file ohasd has been successfully linked to the RC directories
    2010-07-12 03:12:42: Starting ohasd
    2010-07-12 03:12:42: itab entries=
    2010-07-12 03:12:47: Created backup /etc/inittab.no_crs
    2010-07-12 03:12:47: Appending to /etc/inittab.tmp:
    2010-07-12 03:12:47: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
    2010-07-12 03:12:47: Done updating /etc/inittab.tmp
    2010-07-12 03:12:47: Saved /etc/inittab.crs
    2010-07-12 03:12:47: Installed new /etc/inittab
    2010-07-12 03:13:03: ohasd is starting
    2010-07-12 03:13:03: Checking ohasd
    2010-07-12 03:13:03: ohasd started successfully
    2010-07-12 03:13:03: Creating CRS resources and dependencies
    2010-07-12 03:13:03: Configuring HASD
    2010-07-12 03:13:03: Registering type ora.daemon.type
    2010-07-12 03:13:04: Registering type ora.mdns.type
    2010-07-12 03:13:04: Registering type ora.gpnp.type
    2010-07-12 03:13:04: Registering type ora.gipc.type
    2010-07-12 03:13:05: Registering type ora.cssd.type
    2010-07-12 03:13:05: Registering type ora.cssdmonitor.type
    2010-07-12 03:13:06: Registering type ora.crs.type
    2010-07-12 03:13:06: Registering type ora.evm.type
    2010-07-12 03:13:07: Registering type ora.ctss.type
    2010-07-12 03:13:07: Registering type ora.asm.type
    2010-07-12 03:13:08: Registering type ora.drivers.acfs.type
    2010-07-12 03:13:08: Registering type ora.diskmon.type
    2010-07-12 03:13:35: ADVM/ACFS is configured
    2010-07-12 03:13:35: Successfully created CRS resources for cluster daemon and ASM
    2010-07-12 03:13:35: Checking if initial configuration has been performed
    2010-07-12 03:13:35: Starting CSS in exclusive mode
    2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.gipcd' on 'lodvmrhn3'
    2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.mdnsd' on 'lodvmrhn3'
    2010-07-12 03:14:04: CRS-2676: Start of 'ora.mdnsd' on 'lodvmrhn3' succeeded
    2010-07-12 03:14:04: CRS-2676: Start of 'ora.gipcd' on 'lodvmrhn3' succeeded
    2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.gpnpd' on 'lodvmrhn3'
    2010-07-12 03:14:04: CRS-2676: Start of 'ora.gpnpd' on 'lodvmrhn3' succeeded
    2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lodvmrhn3'
    2010-07-12 03:14:04: CRS-2676: Start of 'ora.cssdmonitor' on 'lodvmrhn3' succeeded
    2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.cssd' on 'lodvmrhn3'
    2010-07-12 03:14:04: CRS-2672: Attempting to start 'ora.diskmon' on 'lodvmrhn3'
    2010-07-12 03:14:04: CRS-2676: Start of 'ora.diskmon' on 'lodvmrhn3' succeeded
    2010-07-12 03:14:04: CRS-2676: Start of 'ora.cssd' on 'lodvmrhn3' succeeded
    2010-07-12 03:14:04: Querying for existing CSS voting disks
    2010-07-12 03:14:05: Performing initial configuration for cluster
    2010-07-12 03:14:06: Start of resource "ora.ctssd -init" Succeeded
    2010-07-12 03:14:06: Creating or upgrading OCR keys
    2010-07-12 03:14:06: Command return code of 255 (65280) from command: /u01/app/11.2.0/grid/bin/ocrconfig -upgrade oracle oinstall
    2010-07-12 03:14:06: Failed to create Oracle Cluster Registry configuration, rc 255
    2010-07-12 03:14:06: Exiting exclusive mode
    2010-07-12 03:14:06: Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
    2010-07-12 03:14:06: Stop of resource "ora.crsd -init" failed
    2010-07-12 03:14:06: Failed to stop CRSD
    2010-07-12 03:14:34: Initial cluster configuration failed. See /u01/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_lodvmrhn3.log for details
    Please hel me to resole the issue.
    I have followed exact steps mentioned in the http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnLinuxUsingNFS.php

  • How do I define 2 disk groups for ocr and voting disks at the oracle grid infrastructure installation window

    Hello,
    It may sound too easy to someone but I need to ask it anyway.
    I am in the middle of building Oracle RAC 11.2.0.3 on Linux. I have 2 storage and I created three LUNs in each of storage so 6 LUNs in total look in both of servers. If I choose NORMAL as a redundancy level, is it right to choose my 6 disks those are created for OCR_VOTE at the grid installation? Or should I choose only 3 of them and configure mirroring at latter stage?
    The reason why I am asking this is that I didn't find any place to create asm disk group for ocr and voting disks at the oracle grid infrastructure installation window. In fact, I would to like to create two disk groups in which one of groups contains three disks those were brought from storage 1 and the second one contains rest three 3 disk that come from the last storage.
    I believe that you will understand the manner and help me on choosing proper way.
    Thank you.

    Hi,
    You have 2 Storage H/W.
    You will need to configure a Quorum ASMDisk to store a Voting Disk.
    Because if you lose 1/2 or more of all of your voting disks, then nodes get evicted from the cluster, or nodes kick themselves out of the cluster.
    You must have the odd number of voting disk (i.e 1,3 or 5)  one voting disk per ASM DISK, So 1 Storage will hold more voting disk than another storage.
    (e.g 3 Voting Disk - 2 voting stored in stg-A and 1 voting disk store in stg-B)
    If fail the storage that hold the major number of  Voting disk the whole cluster (all nodes) goes down. Does not matter if you have a other storage online.
    It will happen:
    https://forums.oracle.com/message/9681811#9681811
    You must configure your Clusterware same as RAC extended
    Check this link:
    http://www.oracle.com/technetwork/database/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf
    Explaining: How to store OCR, Voting disks and ASM SPFILE on ASM Diskgroup (RAC or RAC Extended) | Levi Pereira | Oracl…

  • 11.2 Standalone Grid Infrastructure installation to Clustered Grid Insall

    Quick question:
    Currently we have a 11.2.0.1 standalone grid infrastructure installation with ASM instance running on server a1. And there are few databases running under the +ASM instance on a1.  We are planning to make a1, a2 as a 2-node real application cluster.   Is there a way to convert the standalone grid infrastructure installation on server a1 to a clustered grid infrastructure installation along with a2 with out de-installing the existing standalone grid infrastructure on a1 and re-installing freshly?
    Thanks

    Hi Boochi,
    Thanks for responding. May be I am not clear enough in my post. Database conversion will be the next step, my question is about the actual grid infrastructure installation...Currently a1 server has been configured to work as standalone grid infrastructure, but our requirement is to convert to a 2-node RAC with server a2. Generally, I would go by de-installing the existing standalone grid infrastructure running on a1 and re-install the clustered-grid infrastructure with nodes a1 and a2. I am wondering if there is a way without de-installing the existing standalone grid infrastructure??

  • Oracle Grid infrastructure installation tutorial

    Dear friends who can help me by providing some link for Oracle Grid infrastructure installation
    tutorial. I need installation procedure description with GNS, SCAN as well as DNS and DHCP
    configuration for Grid Infra. I already searched in NET and can't found same exhaustive instructions.
    I just want to compare my and others installations ways.
    I hope see your comments soon.
    thanks in advance

    ora_tech wrote:
    Hi,
    Refer blog post by Guenadi Jilevski:
    http://gjilevski.wordpress.com/2011/10/05/build-two-node-oracle-rac-11gr2-11-2-0-3-with-gns-dns-dhcp-and-haip/
    This might be helpful :)
    thanks,
    X A H E E Rthanks for quick answer :)

  • Oracle grid infrastructure installation take long time on step 15 (perform prerequisite checks )

    Oracle grid infrastructure installation take long time on step 15 (perform prerequisite checks ) 
    I'm starting install and configure Oracle RAC 11gR2 on redhat 6.3 , when i reached step 15 (perform prerequisite checks ) it's taking long time more than 10 hours and still checking .... i closed it and start installation again it's same still waiting in perform prerequisite checks 75% complete
    it is normal ?? what should i do ?? how i can monitoring installation steps and why this step take all this time please help .

    Pl identify which exact version of 11gR2 - currently only 11.2.0.3 is certified/supported on Linux 6.x - Oracle&amp;reg; Database - if you are using 11.2.0.1 or 11.2.0.2, then you will need to use Linux 5.x
    HTH
    Srini

  • Blog Post: Grid Infrastructure installation for Standalone

    I hope This document should be useful for who trying to install Grid infrastructure for Standalone server..., Expect comments Also.Read it Enjoy it.
    http://mohamedazar.wordpress.com/2010/11/29/grid-infrastructure-installation-guide-for-standalone-server/

    Please check this metalink note:
    OHASD Failed to Start: Inappropriate ioctl for device (Doc ID 1069182.1)

  • Install a grid infrastructure for a cluster for single node Windows Server

    Hello,
    Can you suggest me how to install a Oracle Single Node RAC 11g on a Windows Server 2008 x64 bits? It's for testing purposes and I don't need two nodes. I know that Linux is better, but my company only uses Windows servers. Our former consultant was able to do it, but didn't tell us how to and that server was lost.
    I know that we have to install first Grid Infrastructure. Running the "Oracle Grid Infrastructure" installer, I could successfully install a "Grid Infrastructure for a Standalone Server". But that grid is not for a RAC database. When trying to install the Database software, it allowed to me to only install "Single instance database", not "Real Application Cluster database installation".
    Running the "Oracle Grid Infrastructure" installer, I tried to select option:
    "Install and Configure Grid Infrastructure for a Cluster" -> Typical Installation:
    SCAN Name: SRVORAC-cluster
    Validating SCAN information
    INS-40922 Invalid SCAN Name - unresolvable to IP address.
    Is it possible to install a grid infrastructure for a cluster for a single-node or one-node Windows Server ?
    Edited by: user521219 on Jan 30, 2012 12:46 PM

    HI,
    You probably have misconception about Single node RAC. This feature does not need that you have only one physical machine in a RAC, it actually means that in a RAC environment (2 or more nodes), you can run your database on only one node(single instance) and then later you can move this single instance to any other node of the RAC (so that you can do some patching on this node making your database available all the time). This also has some other benefits, see bellow.
    http://www.oracle.com/technetwork/database/clustering/overview/ug-raconenode-2009-130760.pdf
    http://docs.oracle.com/cd/E11882_01/install.112/e25666/whatsnew.htm#sthref8
    Salman

  • Oracle 11g R2 Grid Infrastructure Installation stuck on 65%

    Dear All,
    I am in process of installing Oracle 11g R2 Grid Infrastructure on 2-node virtual RAC. All went very well until the Grid Installation stuck on 65%. It is saying copying file on remote node ('Copying Oracle Home '/u01/app/11.2.0/grid' to remote nodes 'cxm-vl-rman02'). From the logs it is difficult to figure out where the bottlenecks lies. Have anyone of you encounter similar situation before? I can attach oraInventory logs as well if you want to have a look at that. Please let me know if you find any pointer or clue of this problem.
    Many Thanks
    Nukesh

    No nothing was being copied, the /u01/app/11.2.0/grid directory was empty and nothing was adding to the log or err files.
    Actually I found/resolved the problem and forgot about this post.
    It happens that in staging my server, iptables was enabled which stopped the progress.
    I had to issue 'service iptables stop' and 'chkconfig iptables off' AND REBOOT ALL NODES.
    Simply stopping the service allowed the installer to continue, but eventually the root.sh script has problems. Rebooting both nodes resolved the issue.
    Thanks

  • What happens after root.sh - grid infrastructure installation

    Hi,
    I'm trying to install 11.2.0.2 grid infrastructure.
    I got ok until root.sh, during root.sh I got some error on the asm disks for the ocr - I think it's a permissions issue.
    Anyway - sysadmin fixed it and I know I now need to deconfigure root.sh -force and then run it root.sh again.
    however - sysadmin also reboot my server so - the installer is GONE.
    My question is - let say I get the root.sh run smothly, what is the installer doing after I press ok on the root.sh screen (which I now of course don't have...)
    can this be run in some script as well?
    Thanks!

    just to close this thread - the answer I gor from oracle support was:
    Hello,
    The OUI will execute cluvfy after root.sh is acknowledge to be run successfully. You can manually execute cluvfy from grid_home with -post option to validate the installation.
    check also with commands that all resources are online:
    crsctl stat res -init
    crsctl stat res -t
    Thanks and regards,

  • Root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1

    Hello all,
    root.sh fails with the errors below. SR with Oracle opened. Will post the resolution when it is available. Any insights in the meantime? Thank you!
    System information:
    OS: AIX 6.1
    Runcluvfy.sh reported no issue
    Permissions on the raw devices set to 660 and ownership is oracle:dba
    Using external redundancy for ASM, ASM instance is online
    Permissions on block and raw device files
    system1:ux460p1> ls -l /dev/hdisk32
    brw-rw---- 1 oracle dba 17, 32 Mar 11 16:50 /dev/hdisk32
    system11:ux460p1> ls -l /dev/rhdisk32
    crw-rw---- 1 oracle dba 17, 32 Mar 12 15:52 /dev/rhdisk32
    ocrconfig.log
    racle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2010-03-15 19:17:19.773: [ OCRCONF][1]ocrconfig starts...
    2010-03-15 19:17:19.775: [ OCRCONF][1]Upgrading OCR data
    2010-03-15 19:17:20.474: [  OCRASM][1]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
    2010-03-15 19:17:20.474: [  OCRASM][1]proprasmo: Error in open/create file in dg [DATA]
    [  OCRASM][1]SLOS : [clsuSlosFormatDiag called with non-error slos.]
    2010-03-15 19:17:20.603: [  OCRRAW][1]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
    2010-03-15 19:17:20.603: [  OCRRAW][1]proprioo: No OCR/OLR devices are usable
    2010-03-15 19:17:20.603: [  OCRASM][1]proprasmcl: asmhandle is NULL
    2010-03-15 19:17:20.603: [  OCRRAW][1]proprinit: Could not open raw device
    2010-03-15 19:17:20.603: [  OCRASM][1]proprasmcl: asmhandle is NULL
    2010-03-15 19:17:20.604: [ default][1]a_init:7!: Backend init unsuccessful : [26]
    2010-03-15 19:17:20.604: [ OCRCONF][1]Exporting OCR data to [OCRUPGRADEFILE]
    2010-03-15 19:17:20.604: [  OCRAPI][1]a_init:7!: Backend init unsuccessful : [33]
    2010-03-15 19:17:20.605: [ OCRCONF][1]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
    2010-03-15 19:17:20.841: [  OCRASM][1]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
    2010-03-15 19:17:20.841: [  OCRASM][1]proprasmo: Error in open/create file in dg [DATA]
    [  OCRASM][1]SLOS : [clsuSlosFormatDiag called with non-error slos.]
    2010-03-15 19:17:20.966: [  OCRRAW][1]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
    2010-03-15 19:17:20.966: [  OCRRAW][1]proprioo: No OCR/OLR devices are usable
    2010-03-15 19:17:20.966: [  OCRASM][1]proprasmcl: asmhandle is NULL
    2010-03-15 19:17:20.966: [  OCRRAW][1]proprinit: Could not open raw device
    2010-03-15 19:17:20.966: [  OCRASM][1]proprasmcl: asmhandle is NULL
    2010-03-15 19:17:20.966: [ default][1]a_init:7!: Backend init unsuccessful : [26]
    2010-03-15 19:17:21.412: [  OCRRAW][1]propriogid:1_2: INVALID FORMAT
    2010-03-15 19:17:21.412: [  OCRRAW][1]proprior: Header check from OCR device 0 offset 0 failed (26).
    2010-03-15 19:17:21.414: [  OCRRAW][1]ibctx: Failed to read the whole bootblock. Assumes invalid format.
    2010-03-15 19:17:21.414: [  OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22
    2010-03-15 19:17:21.534: [  OCRRAW][1]propriogid:1_2: INVALID FORMAT
    2010-03-15 19:17:21.701: [  OCRRAW][1]iniconfig:No 92 configuration
    2010-03-15 19:17:21.701: [  OCRAPI][1]a_init:6a: Backend init successful
    2010-03-15 19:17:21.764: [ OCRCONF][1]Initialized DATABASE keys
    2010-03-15 19:17:21.770: [ OCRCONF][1]Successfully set skgfr block 0
    2010-03-15 19:17:21.771: [ OCRCONF][1]Exiting [status=success]...
    **alert.log**
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2010-03-15 19:12:00.148
    [client(483478)]CRS-2106:The OLR location /u01/app/grid/cdata/ux460p1.olr is inaccessible. Details in /u01/app/grid/log/ux460p1/client/ocrconfig_483478.log.
    2010-03-15 19:12:00.171
    [client(483478)]CRS-2101:The OLR was formatted using version 3.
    2010-03-15 14:16:18.620
    [ohasd(471204)]CRS-2112:The OLR service started on node ux460p1.
    2010-03-15 14:16:18.720
    [ohasd(471204)]CRS-8017:location: /etc/oracle/lastgasp has 8 reboot advisory log files, 0 were announced and 0 errors occurred
    2010-03-15 14:16:18.847
    [ohasd(471204)]CRS-2772:Server 'ux460p1' has been assigned to pool 'Free'.
    2010-03-15 14:16:54.107
    [ctssd(340174)]CRS-2403:The Cluster Time Synchronization Service on host ux460p1 is in observer mode.
    2010-03-15 14:16:54.123
    [ctssd(340174)]CRS-2407:The new Cluster Time Synchronization Service reference node is host ux460p1.
    2010-03-15 14:16:54.917
    [ctssd(340174)]CRS-2401:The Cluster Time Synchronization Service started on host ux460p1.
    2010-03-15 19:17:21.414
    [client(376968)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/grid/log/ux460p1/client/ocrconfig_376968.log.
    2010-03-15 19:17:21.701
    [client(376968)]CRS-1001:The OCR was formatted using version 3.
    2010-03-15 14:17:24.888
    [crsd(303252)]CRS-1012:The OCR service started on node ux460p1.
    2010-03-15 14:17:56.344
    [ctssd(340174)]CRS-2405:The Cluster Time Synchronization Service on host ux460p1 is shutdown by user
    2010-03-15 14:19:14.855
    [ctssd(340188)]CRS-2403:The Cluster Time Synchronization Service on host ux460p1 is in observer mode.
    2010-03-15 14:19:14.870
    [ctssd(340188)]CRS-2407:The new Cluster Time Synchronization Service reference node is host ux460p1.
    2010-03-15 14:19:15.638
    [ctssd(340188)]CRS-2401:The Cluster Time Synchronization Service started on host ux460p1.
    2010-03-15 14:19:32.985
    [crsd(417946)]CRS-1012:The OCR service started on node ux460p1.
    2010-03-15 14:19:35.250
    [crsd(417946)]CRS-1201:CRSD started on node ux460p1.
    2010-03-15 14:19:35.698
    [ohasd(471204)]CRS-2765:Resource 'ora.crsd' has failed on server 'ux460p1'.
    2010-03-15 14:19:38.928

    Public and Private are on different devices and subnets.
    No logfile named: ocrconfig_7833.log
    I do have ocrconfig_7089.log and ocrconfig_8985.log
    Here is their contents:
    ocrconfig_7089.log:
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2010-11-09 13:38:32.518: [ OCRCONF][2819644944]ocrconfig starts...
    2010-11-09 13:38:32.542: [ OCRCONF][2819644944]Upgrading OCR data
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
    2010-11-09 13:38:32.576: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:38:32.576: [  OCRRAW][2819644944]proprioini: all disks are not OCR/OLR formatted
    2010-11-09 13:38:32.576: [  OCRRAW][2819644944]proprinit: Could not open raw device
    2010-11-09 13:38:32.576: [ default][2819644944]a_init:7!: Backend init unsuccessful : [26]
    2010-11-09 13:38:32.577: [ OCRCONF][2819644944]Exporting OCR data to [OCRUPGRADEFILE]
    2010-11-09 13:38:32.577: [  OCRAPI][2819644944]a_init:7!: Backend init unsuccessful : [33]
    2010-11-09 13:38:32.577: [ OCRCONF][2819644944]There was no previous version of OCR. error:[PROCL-33: Oracle Local Registry is not configured]
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
    2010-11-09 13:38:32.578: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:38:32.578: [  OCRRAW][2819644944]proprioini: all disks are not OCR/OLR formatted
    2010-11-09 13:38:32.578: [  OCRRAW][2819644944]proprinit: Could not open raw device
    2010-11-09 13:38:32.578: [ default][2819644944]a_init:7!: Backend init unsuccessful : [26]
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
    2010-11-09 13:38:32.579: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
    2010-11-09 13:38:32.591: [  OCRRAW][2819644944]ibctx: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:38:32.591: [  OCRRAW][2819644944]proprinit:problem reading the bootblock or superbloc 22
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 0
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 1
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 2
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 3
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 4
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 5
    2010-11-09 13:38:32.591: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:38:32.681: [  OCRAPI][2819644944]a_init:6a: Backend init successful
    2010-11-09 13:38:32.699: [ OCRCONF][2819644944]Initialized DATABASE keys
    2010-11-09 13:38:32.700: [ OCRCONF][2819644944]Exiting [status=success]...
    ocrconfig_8985.log:
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2010-11-09 13:41:28.169: [ OCRCONF][2281741840]ocrconfig starts...
    2010-11-09 13:41:28.175: [ OCRCONF][2281741840]Upgrading OCR data
    2010-11-09 13:41:30.896: [  OCRASM][2281741840]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
    2010-11-09 13:41:30.896: [  OCRASM][2281741840]proprasmo: Error in open/create file in dg [DATA]
    [  OCRASM][2281741840]SLOS : [clsuSlosFormatDiag called with non-error slos.]
    2010-11-09 13:41:31.208: [  OCRRAW][2281741840]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
    2010-11-09 13:41:31.210: [  OCRRAW][2281741840]proprioo: No OCR/OLR devices are usable
    2010-11-09 13:41:31.210: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
    2010-11-09 13:41:31.210: [  OCRRAW][2281741840]proprinit: Could not open raw device
    2010-11-09 13:41:31.211: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
    2010-11-09 13:41:31.213: [ default][2281741840]a_init:7!: Backend init unsuccessful : [26]
    2010-11-09 13:41:31.214: [ OCRCONF][2281741840]Exporting OCR data to [OCRUPGRADEFILE]
    2010-11-09 13:41:31.216: [  OCRAPI][2281741840]a_init:7!: Backend init unsuccessful : [33]
    2010-11-09 13:41:31.216: [ OCRCONF][2281741840]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
    2010-11-09 13:41:32.214: [  OCRASM][2281741840]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
    2010-11-09 13:41:32.214: [  OCRASM][2281741840]proprasmo: Error in open/create file in dg [DATA]
    [  OCRASM][2281741840]SLOS : [clsuSlosFormatDiag called with non-error slos.]
    2010-11-09 13:41:32.535: [  OCRRAW][2281741840]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
    2010-11-09 13:41:32.535: [  OCRRAW][2281741840]proprioo: No OCR/OLR devices are usable
    2010-11-09 13:41:32.535: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
    2010-11-09 13:41:32.535: [  OCRRAW][2281741840]proprinit: Could not open raw device
    2010-11-09 13:41:32.535: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
    2010-11-09 13:41:32.536: [ default][2281741840]a_init:7!: Backend init unsuccessful : [26]
    2010-11-09 13:41:35.359: [  OCRRAW][2281741840]propriogid:1_2: INVALID FORMAT
    2010-11-09 13:41:35.361: [  OCRRAW][2281741840]proprior: Header check from OCR device 0 offset 0 failed (26).
    2010-11-09 13:41:35.363: [  OCRRAW][2281741840]ibctx: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:41:35.363: [  OCRRAW][2281741840]proprinit:problem reading the bootblock or superbloc 22
    2010-11-09 13:41:35.843: [  OCRRAW][2281741840]propriogid:1_2: INVALID FORMAT
    2010-11-09 13:41:36.430: [  OCRRAW][2281741840]iniconfig:No 92 configuration
    2010-11-09 13:41:36.431: [  OCRAPI][2281741840]a_init:6a: Backend init successful
    2010-11-09 13:41:36.540: [ OCRCONF][2281741840]Initialized DATABASE keys
    2010-11-09 13:41:36.545: [ OCRCONF][2281741840]Successfully set skgfr block 0
    2010-11-09 13:41:36.552: [ OCRCONF][2281741840]Exiting [status=success]...
    Both of these log files show errors, then they show success??????

  • Grid installation: root.sh failed on the first node on Solaris cluster 4.1

    Hi all,
    I'm trying to install the Grid (11.2.0.3.0) on the 2 node-clusters (OSC 4.1).
    When I run the root.sh on the first node, I got the out put as follow:
    xha239080-root-5.11# root.sh
    Performing root user operation for Oracle 11g
    The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /Grid/CRShome
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
    /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]:
    Warning: /usr/local/bin is read only. No files will be copied.
    Creating /var/opt/oracle/oratab file...
    Entries will be added to the /var/opt/oracle/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root script.
    Now product-specific root actions will be performed.
    Using configuration parameter file: /Grid/CRShome/crs/install/crsconfig_params
    Creating trace directory
    User ignored Prerequisites during installation
    OLR initialization - successful
    root wallet
    root wallet cert
    root cert export
    peer wallet
    profile reader wallet
    pa wallet
    peer wallet keys
    pa wallet keys
    peer cert request
    pa cert request
    peer cert
    pa cert
    peer root cert TP
    profile reader root cert TP
    pa root cert TP
    peer pa cert TP
    pa peer cert TP
    profile reader pa cert TP
    profile reader peer cert TP
    peer user cert
    pa user cert
    Adding Clusterware entries to inittab
    CRS-2672: Attempting to start 'ora.mdnsd' on 'xha239080'
    CRS-2676: Start of 'ora.mdnsd' on 'xha239080' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'xha239080'
    CRS-2676: Start of 'ora.gpnpd' on 'xha239080' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'xha239080'
    CRS-2672: Attempting to start 'ora.gipcd' on 'xha239080'
    CRS-2676: Start of 'ora.cssdmonitor' on 'xha239080' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'xha239080' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'xha239080'
    CRS-2672: Attempting to start 'ora.diskmon' on 'xha239080'
    CRS-2676: Start of 'ora.diskmon' on 'xha239080' succeeded
    CRS-2676: Start of 'ora.cssd' on 'xha239080' succeeded
    ASM created and started successfully.
    Disk Group DATA created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    CRS-4256: Updating the profile
    Successful addition of voting disk 9cdb938773bc4f16bf332edac499fd06.
    Successful addition of voting disk 842907db11f74f59bf65247138d6e8f5.
    Successful addition of voting disk 748852d2a5c84f72bfcd50d60f65654d.
    Successfully replaced voting disk group with +DATA.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ## STATE File Universal Id File Name Disk group
    1. ONLINE 9cdb938773bc4f16bf332edac499fd06 (/dev/did/rdsk/d10s6) [DATA]
    2. ONLINE 842907db11f74f59bf65247138d6e8f5 (/dev/did/rdsk/d8s6) [DATA]
    3. ONLINE 748852d2a5c84f72bfcd50d60f65654d (/dev/did/rdsk/d9s6) [DATA]
    Located 3 voting disk(s).
    Start of resource "ora.cssd" failed
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'xha239080'
    CRS-2672: Attempting to start 'ora.gipcd' on 'xha239080'
    CRS-2676: Start of 'ora.cssdmonitor' on 'xha239080' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'xha239080' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'xha239080'
    CRS-2672: Attempting to start 'ora.diskmon' on 'xha239080'
    CRS-2676: Start of 'ora.diskmon' on 'xha239080' succeeded
    CRS-2674: Start of 'ora.cssd' on 'xha239080' failed
    CRS-2679: Attempting to clean 'ora.cssd' on 'xha239080'
    CRS-2681: Clean of 'ora.cssd' on 'xha239080' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'xha239080'
    CRS-2677: Stop of 'ora.gipcd' on 'xha239080' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'xha239080'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'xha239080' succeeded
    CRS-5804: Communication error with agent process
    CRS-4000: Command Start failed, or completed with errors.
    Failed to start Oracle Grid Infrastructure stack
    Failed to start Cluster Synchorinisation Service in clustered mode at /Grid/CRShome/crs/install/crsconfig_lib.pm line 1211.
    /Grid/CRShome/perl/bin/perl -I/Grid/CRShome/perl/lib -I/Grid/CRShome/crs/install /Grid/CRShome/crs/install/rootcrs.pl execution failed
    xha239080-root-5.11# history
    checking the ocssd.log, I see some thing as follow:
    2013-09-16 18:46:24.238: [    CSSD][1]clssscmain: Starting CSS daemon, version 11.2.0.3.0, in (clustered) mode with uniqueness value 1379371584
    2013-09-16 18:46:24.239: [    CSSD][1]clssscmain: Environment is production
    2013-09-16 18:46:24.239: [    CSSD][1]clssscmain: Core file size limit extended
    2013-09-16 18:46:24.248: [    CSSD][1]clssscmain: GIPCHA down 1
    2013-09-16 18:46:24.249: [    CSSD][1]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
    2013-09-16 18:46:24.250: [    CSSD][1]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
    2013-09-16 18:46:24.250: [    CSSD][1]clssscExtendLimits: The current soft limit for locked memory is 4294967293, hard limit is 4294967293
    2013-09-16 18:46:24.250: [    CSSD][1]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
    2013-09-16 18:46:24.250: [    CSSD][1]clssscSetPrivEnv: Setting priority to 4
    2013-09-16 18:46:24.253: [    CSSD][1]clssscSetPrivEnv: unable to set priority to 4
    2013-09-16 18:46:24.253: [    CSSD][1]SLOS: cat=-2, opn=scls_mem_lockdown, dep=11, loc=mlockall
    unable to lock memory
    2013-09-16 18:46:24.253: [    CSSD][1](:CSSSC00011:)clssscExit: A fatal error occurred during initialization
    Do anyone have any idea what going on and how can I fix it ?

    Hi,
    solaris has several issues with DISM, e.g.:
    Solaris 10 and Solaris 11 Shared Memory Locking May Fail (Doc ID 1590151.1)
    Sounds like Solaris Cluster  has a similar bug. A "workaround" is to reboot the (cluster) zone, that "fixes" the mlock error. This bug was introduced with updates in september, atleast to our environment (Solaris 11.1). Prior i did not have the issue and now i have to restart the entire zone, whenever i stop crs.
    With 11.2.0.3 the root.sh script can be rerun without prior cleaning up, so you should be able to continue installation at that point after the reboot. After the root.sh completes some configuration assistants need to be run, to complete the installation. You need to execute this manually as you wipe your oui session
    Kind Regards
    Thomas

  • Troubles with Grid Infrastructure Installation

    Hello everybody
    Enviroment:
    O.S: SLES SP1 x64
    Oracle 11g R2 Update 3
    I want to install this product and make a RAC of tow nodes. Since some days ago I have a problem with the installation process. When I perform ssh connectivity setup, this errors appears at ssh log configuration:
    INFO: [rac2.mydomain.com]
    INFO: /bin/mkdir -p "/opt/oracle/.ssh"
    INFO: Exit-status: 0
    INFO: Error:
    INFO:
    INFO:
    INFO: [rac2.mydomain.com]
    INFO: /bin/rm -f "/opt/oracle/.ssh/id_rsa.pubrac2.mydomain.com"
    INFO: Exit-status: 0
    INFO: Error:
    INFO:
    INFO:
    INFO: Copy localfile: /opt/oracle/.ssh/id_rsa.pub to remotefile: "/opt/oracle/.ssh/id_rsa.pubrac2.mydomain.com"
    INFO: [rac2.mydomain.com]
    INFO: /bin/cp -p "/opt/oracle/.ssh/authorized_keys" "/opt/oracle/.ssh/authorized_keys.ri.bak"
    INFO: Exit-status: 1
    INFO: Error: /bin/cp: cannot stat `/opt/oracle/.ssh/authorized_keys': No such file or directory
    INFO: Creating keys on remote Host rac1.mydomain.com
    INFO: to setup SSH on Host rac1.mydomain.com
    INFO: [rac1.mydomain.com]
    INFO: /bin/rm /opt/oracle/.ssh/identity.pub
    INFO: Exit-status: 1
    INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity.pub': No such file or directory
    INFO:
    INFO:
    INFO: [rac1.mydomain.com]
    INFO: /bin/rm /opt/oracle/.ssh/identity
    INFO: Exit-status: 1
    INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity': No such file or directory
    NFO:
    INFO: Creating keys on remote Host rac2.mydomain.com
    INFO: to setup SSH on Host rac2.mydomain.com
    INFO: [rac2.mydomain.com]
    INFO: /bin/rm /opt/oracle/.ssh/identity.pub
    INFO: Exit-status: 1
    INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity.pub': No such file or directory
    INFO:
    INFO:
    INFO: [rac2.mydomain.com]
    INFO: /bin/rm /opt/oracle/.ssh/identity
    INFO: Exit-status: 1
    INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity': No such file or directory
    At installation logs, these lines appear:
    INFO: SSH Setup policy used: BIDIRECTIONAL
    INFO: Preparing to check passwordless SSH Connectivity between nodes: [rac1, rac2]
    INFO: Testing passwordless SSH connectivity between the selected nodes. This may take several minutes, please wait...
    INFO: OverallStatus of User Equivalence check using CVU is SUCCESSFUL
    INFO: Finding passwordless SSH connectivity setup on remote nodes
    INFO: Preparing remote commands to check passwordless SSH connectivity setup on remote nodes
    INFO: Total number of remote commands to be executed is 2
    WARNING: Failed while executing remote commands to check SSH connectivity to other nodes..
    Refer associated stacktrace #oracle.install.driver.oui.OUISSHSupportManager:887
    INFO: Source Node: rac2 Target Node: rac1
    INFO: Exception: null
    INFO: Status log: oracle.install.driver.oui.OUISSHSupportManager$SSHSetupVerificationCommand: 0|Permission denied (publickey,keyboard-interac
    tive). :failed
    INFO: Source Node: rac2 Target Node: rac2
    INFO: Exception: null
    INFO: Status log: oracle.install.driver.oui.OUISSHSupportManager$SSHSetupVerificationCommand: 0|Permission denied (publickey,keyboard-interac
    tive). :failed
    INFO: Remote node(s) not configured for passwordless SSH connectivity : [rac2]
    INFO: Details about the list of remote node(s), not configured for passwordless SSH connectivity
    NODE REMOTE NODES
    rac2 [rac1, rac2]
    INFO: Pause Oracle Grid Infrastructure
    INFO: Resume Oracle Grid Infrastructure
    INFO: Adding ExitStatus USER_CANCELLED_INSTALL to the exit status set
    INFO: Finding the most appropriate exit status for the current application
    INFO: Exit Status is 4
    INFO: Shutdown Oracle Grid Infrastructure
    INFO: Unloading Setup Driver
    To confirm that everything is going wrong, when I click in Test button, this error appears:
    [INS_06006] Passwordless SSH connectivity not set up between the following node(s): [rac2]
    I can do ssh from rac1 to rac2 without password, but I can´t from rac2 to rac1
    I run cluvfy.sh and everything is ok.
    I hope for helping.
    Thanks a lot

    The problem begins when on remote node It can´t perform a several operations as logs appears.
    INFO: [rac2.mydomain.com]
    INFO: /bin/cp -p "/opt/oracle/.ssh/authorized_keys" "/opt/oracle/.ssh/authorized_keys.ri.bak"
    INFO: Exit-status: 1
    INFO: Error: /bin/cp: cannot stat `/opt/oracle/.ssh/authorized_keys': No such file or directory
    INFO: Creating keys on remote Host rac1.mydomain.com
    INFO: to setup SSH on Host rac1.mydomain.com
    INFO: [rac1.mydomain.com]
    INFO: /bin/rm /opt/oracle/.ssh/identity.pub
    INFO: Exit-status: 1
    INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity.pub': No such file or directory
    INFO:
    INFO:
    INFO: [rac1.mydomain.com]
    INFO: /bin/rm /opt/oracle/.ssh/identity
    INFO: Exit-status: 1
    INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity': No such file or directory
    NFO:
    INFO: Creating keys on remote Host rac2.mydomain.com
    INFO: to setup SSH on Host rac2.mydomain.com
    INFO: [rac2.mydomain.com]
    INFO: /bin/rm /opt/oracle/.ssh/identity.pub
    INFO: Exit-status: 1
    INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity.pub': No such file or directory
    INFO:
    INFO:
    INFO: [rac2.mydomain.com]
    INFO: /bin/rm /opt/oracle/.ssh/identity
    INFO: Exit-status: 1
    INFO: Error: /bin/rm: cannot remove `/opt/oracle/.ssh/identity': No such file or directory

  • Grid Infrastructure installable for Oracle Restart only (No ASM)?

    Hello,
    Can Oracle Grid Infrastructure be installed for its Oracle Restart functionality only? i.e., No ASM,; this box uses local RAID 5 filesystems only.
    This is a standalone 32-bit RHEL 5 system with Oracle Database 10gR2 already installed that I intend to upgrade to 11gR2. Just experimenting with installing Grid Infrastructure for only its restart capability. There isn't anything out there I've found that addresses this other than the Oracle doc stating that the Infrastructure must be installed if I want to use ASM or Oracle Restart.
    I'm currently in the installer in a Software Only installation waiting for my SysAdmin to run the root.sh script, but in looking at this script, it looks like it might try to do things with CRS, voting files and the like, which do not exist.
    Thank you!
    -Jon

    Hi Erik,
    Can Oracle Grid Infrastructure be installed for its Oracle Restart functionality only? Yes ... you can do that installing Software Only
    I'm currently in the installer in a Software Only installation waiting for my SysAdmin to run the root.sh script, but in looking at this script, it looks like it might try to do things with CRS, voting files and the like, which do not exist.Don't worry about that ... at the end of the execution of the root.sh script will report a proceedings to configure only Oracle Restart.
    Only this resource "ora.evmd" will appear online.
    Regards,
    Levi Pereira

Maybe you are looking for

  • Dynamic visual attributes/multi-row blocks

    We're using Headstart, Des2k V6.0.3.8. If you set preference CURREC = multi record block, you get the current record visual attribute (cg$current_record) attached to all displayed items in multi-row blocks. From the Headstart object library, these it

  • Downloading and installing applications

    I can download applications via the web browser but cannot get them to install - specifically, I am trying to install FLIP4MAC so I can play Windows Media files via Quicktime. I download to the desktop and when I double click to install it keeps aski

  • I have an error message ox61011bed on my hp photosmart plus b209a-m. what does this mean?

    i have an error message ox61011bed on my hp photosmart plus b209a-m. what does this mean? The printer was working fine last night but this morning does not want to print.

  • Running Form " Browser Problem "

    hiiii everybody ... my problem is that when trying to run a form the browser starts and before it loads the form the browser close automaticaly ... this is anew problem i dont know what are the possible causes for it ... Hope too find any help here .

  • Getting the number of the copy in smartform

    Hi Everyone. I need to know in the smarform the number of the copy that i'm working with. Can anyone help me please. Thanks & Regards   David