Asm 11.1.0.6 root.sh  fails to start css

We have Oracle ASM and installing 11.1.0.6 on a new machine (eventually will apply 11.1.0.7 patchset). So we are on 11g R1 in production environments.
When installing Oracle ASM 11.1.0.6, the root.sh fails
I checked various metalink notes, all settings are OK. We have ASMLib and i have configured ASMLib with the new disks prior to the ASM installation
Just to rule out the rootcause is with ASMLib , I have also disabled it. Still same problem. I ran the usual localconfig delete and localconfig add too.
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.+
Oracle CSS service would not start as installed+
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started+
Finished product-specific root actions.+

I applied the 11.1.0.7 patchset. Still same problem.
Strangely, there are no logfiles in $ASM_HOME/log/<hostname>/cssd/ - This directory is empty
The below are the final messages in root.sh
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started

Similar Messages

  • Failed to start ASM instance "+ASM1" on node "rac1"

    I have a problem, because when I start RAC and write command crs_stat -t
    column State have 2 wrong parameter..
    Name Type Target State Host
    ora.....CRM.cs application ONLINE ONLINE rac2
    ora....db1.srv application ONLINE ONLINE rac2
    ora.devdb.db application ONLINE ONLINE rac2
    ora....b1.inst application ONLINE OFFLINE
    ora....b2.inst application ONLINE ONLINE rac2
    ora....SM1.asm application ONLINE UNKNOWN rac1
    ora....C1.lsnr application ONLINE ONLINE rac1
    ora.rac1.gsd application ONLINE ONLINE rac1
    ora.rac1.ons application ONLINE ONLINE rac1
    ora.rac1.vip application ONLINE ONLINE rac1
    ora....SM2.asm application ONLINE ONLINE rac2
    ora....C2.lsnr application ONLINE ONLINE rac2
    ora.rac2.gsd application ONLINE ONLINE rac2
    ora.rac2.ons application ONLINE ONLINE rac2
    ora.rac2.vip application ONLINE ONLINE rac2
    When I try
    srvctl start asm -n rac1 then is wrong:
    PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac1", [PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac1", [CRS-1028: Dependency analysis failed because of:
    CRS-0223: Resource 'ora.rac1.ASM1.asm' has placement error.]]
    [PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac1", [CRS-1028: Dependency analysis failed because of:
    CRS-0223: Resource 'ora.rac1.ASM1.asm' has placement error.]]
    and when I try start instance manualy then
    PRKP-1001 : Error starting instance devdb1 on node rac1
    CRS-1028: Dependency analysis failed because of:
    CRS-0223: Resource 'ora.devdb.devdb1.inst' has placement error.
    :( Where is my problem??

    hi, i have exactly the same error
    but your suggestions of remove an recreate the asm resource not working
    ./srvctl remove asm -n dbs2 -i +ASM2 -f
    PRKS-1023 : Failed to remove CRS resource for ASM instance "+ASM2" on node "dbs2", [CRS-0214: Could not unregister resource 'ora.dbs2.ASM2.asm'.]
    ./srvctl start asm -n dbs2
    PRKS-1009 : Failed to start ASM instance "+ASM2" on node "dbs2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "dbs2", [CRS-1028: Dependency analysis failed because of:
    CRS-0223: Resource 'ora.dbs2.ASM2.asm' has placement error.]]
    [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "dbs2", [CRS-1028: Dependency analysis failed because of:
    CRS-0223: Resource 'ora.dbs2.ASM2.asm' has placement error.]]
    how do i proceed?
    iam using solaris 10 with t2000 and t5210 server and oracle 10.2.0.4

  • Root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource

    Dear all,
    I am trying to install clean Oracle 11.2.0.3 grid infrastructure on a two node cluster running on Solaris 5.10.
    - Cluster verification was successfully on both nodes; No warning or issues;
    - I am using 2 network cards for the public and 2 for the private interconnect;
    - OCR is stored on ASM
    - Firewall is disabled on both nodes
    - SCAN is being configured on the DNS (not added in /etc/hosts)
    - GNS is not used
    - hosts file is identical (except the primary hostname)
    The problem: root.sh fails on the 2nd (remote) node, because it fails to start the "ora.asm" resource. However, the root.sh has completed successfully on the 1st node.. Somehow, root.sh doesn't create +ASM2 instance on the remote (host2) node.
    root.sh was executed first on the local node (host1) and after the successful execution was started on the remote (host2) node.
    Output from host1 (working):
    ===================
    Adding Clusterware entries to inittab
    CRS-2672: Attempting to start 'ora.mdnsd' on 'host1'
    CRS-2676: Start of 'ora.mdnsd' on 'host1' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'host1'
    CRS-2676: Start of 'ora.gpnpd' on 'host1' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'host1'
    CRS-2672: Attempting to start 'ora.gipcd' on 'host1'
    CRS-2676: Start of 'ora.cssdmonitor' on 'host1' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'host1' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'host1'
    CRS-2672: Attempting to start 'ora.diskmon' on 'host1'
    CRS-2676: Start of 'ora.diskmon' on 'host1' succeeded
    CRS-2676: Start of 'ora.cssd' on 'host1' succeeded
    ASM created and started successfully.
    Disk Group CRS created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    CRS-4256: Updating the profile
    Successful addition of voting disk 4373be34efab4f01bf79f6c5362acfd3.
    Successful addition of voting disk 7fd725fa4d904f07bf76cecf96791547.
    Successful addition of voting disk a9c85297bdd74f3abfd86899205aaf17.
    Successfully replaced voting disk group with +CRS.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ## STATE File Universal Id File Name Disk group
    1. ONLINE 4373be34efab4f01bf79f6c5362acfd3 (/dev/rdsk/c4t600A0B80006E2CC40000C6674E82AA57d0s4) [CRS]
    2. ONLINE 7fd725fa4d904f07bf76cecf96791547 (/dev/rdsk/c4t600A0B80006E2CC40000C6694E82AADDd0s4) [CRS]
    3. ONLINE a9c85297bdd74f3abfd86899205aaf17 (/dev/rdsk/c4t600A0B80006E2F100000C7744E82AC7Ad0s4) [CRS]
    Located 3 voting disk(s).
    CRS-2672: Attempting to start 'ora.asm' on 'host1'
    CRS-2676: Start of 'ora.asm' on 'host1' succeeded
    CRS-2672: Attempting to start 'ora.CRS.dg' on 'host1'
    CRS-2676: Start of 'ora.CRS.dg' on 'host1' succeeded
    CRS-2672: Attempting to start 'ora.registry.acfs' on 'host1'
    CRS-2676: Start of 'ora.registry.acfs' on 'host1' succeeded
    Configure Oracle Grid Infrastructure for a Cluster ... succeeded
    Name Type Target State Host
    ora.CRS.dg ora....up.type ONLINE ONLINE host1
    ora....ER.lsnr ora....er.type ONLINE ONLINE host1
    ora....N1.lsnr ora....er.type ONLINE ONLINE host1
    ora....N2.lsnr ora....er.type ONLINE ONLINE host1
    ora....N3.lsnr ora....er.type ONLINE ONLINE host1
    ora.asm ora.asm.type ONLINE ONLINE host1
    ora....SM1.asm application ONLINE ONLINE host1
    ora....B1.lsnr application ONLINE ONLINE host1
    ora....db1.gsd application OFFLINE OFFLINE
    ora....db1.ons application ONLINE ONLINE host1
    ora....db1.vip ora....t1.type ONLINE ONLINE host1
    ora.cvu ora.cvu.type ONLINE ONLINE host1
    ora.gsd ora.gsd.type OFFLINE OFFLINE
    ora....network ora....rk.type ONLINE ONLINE host1
    ora.oc4j ora.oc4j.type ONLINE ONLINE host1
    ora.ons ora.ons.type ONLINE ONLINE host1
    ora....ry.acfs ora....fs.type ONLINE ONLINE host1
    ora.scan1.vip ora....ip.type ONLINE ONLINE host1
    ora.scan2.vip ora....ip.type ONLINE ONLINE host1
    ora.scan3.vip ora....ip.type ONLINE ONLINE host1
    Output from host2 (failing):
    ===================
    OLR initialization - successful
    Adding Clusterware entries to inittab
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node billdb1, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    Start of resource "ora.asm" failed
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'host2'
    CRS-2676: Start of 'ora.drivers.acfs' on 'host2' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'host2'
    CRS-5017: The resource action "ora.asm start" encountered the following error:
    ORA-03113: end-of-file on communication channel
    Process ID: 0
    Session ID: 0 Serial number: 0
    *. For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log".*
    CRS-2674: Start of 'ora.asm' on 'host2' failed
    CRS-2679: Attempting to clean 'ora.asm' on 'host2'
    CRS-2681: Clean of 'ora.asm' on 'host2' succeeded
    CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'host2'
    CRS-2677: Stop of 'ora.drivers.acfs' on 'host2' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    Failed to start Oracle Grid Infrastructure stack
    Failed to start ASM at /u01/11.2.0/grid/crs/install/crsconfig_lib.pm line 1272.
    /u01/11.2.0/grid/perl/bin/perl -I/u01/11.2.0/grid/perl/lib -I/u01/11.2.0/grid/crs/install /u01/11.2.0/grid/crs/install/rootcrs.pl execution failed
    Contents of "/u01/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_host2.log"
    =============================================
    CRS-2672: Attempting to start 'ora.asm' on 'host2'
    CRS-5017: The resource action "ora.asm start" encountered the following error:
    ORA-03113: end-of-file on communication channel
    Process ID: 0
    Session ID: 0 Serial number: 0
    . For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log".
    CRS-2674: Start of 'ora.asm' on 'host2' failed
    CRS-2679: Attempting to clean 'ora.asm' on 'host2'
    CRS-2681: Clean of 'ora.asm' on 'host2' succeeded
    CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'host2'
    CRS-2677: Stop of 'ora.drivers.acfs' on 'host2' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    2011-10-24 19:36:54: Failed to start Oracle Grid Infrastructure stack
    2011-10-24 19:36:54: ###### Begin DIE Stack Trace ######
    2011-10-24 19:36:54: Package File Line Calling
    2011-10-24 19:36:54: --------------- -------------------- ---- ----------
    2011-10-24 19:36:54: 1: main rootcrs.pl 375 crsconfig_lib::dietrap
    2011-10-24 19:36:54: 2: crsconfig_lib crsconfig_lib.pm 1272 main::__ANON__
    2011-10-24 19:36:54: 3: crsconfig_lib crsconfig_lib.pm 1171 crsconfig_lib::start_cluster
    2011-10-24 19:36:54: 4: main rootcrs.pl 803 crsconfig_lib::perform_start_cluster
    2011-10-24 19:36:54: ####### End DIE Stack Trace #######
    Shortened output from "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] clean {
    2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] InstAgent::stop_option stop mode immediate option 1
    2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] InstAgent::stop {
    2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] InstAgent::stop original reason system do shutdown abort
    2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] ConnectionPool::resetConnection s_statusOfConnectionMap 00ab1948
    2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] ConnectionPool::resetConnection sid +ASM2 status  2
    2011-10-24 19:35:48.728: [ora.asm][9] {0:0:224} [start] Gimh::check OH /u01/11.2.0/grid SID +ASM2
    2011-10-24 19:35:48.728: [ora.asm][9] {0:0:224} [start] Gimh::check condition changes to (GIMH_NEXT_NUM) 0,1,7 exists
    2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] (:CLSN00006:)AsmAgent::check failed gimh state 0
    2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 0
    2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet { entry
    2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet procr_get_conf: retval [0] configured [1] local only [0] error buffer []
    2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet procr_get_conf: OCR loc [0], Disk Group : [+CRS]
    2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet m_ocrDgpSet 015fba90 dgName CRS
    2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet ocrret 0 found 1
    2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet ocrDgpSet CRS
    2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet exit }
    2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::ocrDgCheck Entry {
    2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] DgpAgent::getConnxn new pool
    2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] DgpAgent::getConnxn new pool m_oracleHome:/u01/11.2.0/grid m_oracleSid:+ASM2 m_usrOraEnv:
    2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] ConnectionPool::ConnectionPool 2 m_oracleHome:/u01/11.2.0/grid, m_oracleSid:+ASM2, m_usrOraEnv:
    2011-10-24 19:35:48.733: [ora.asm][9] {0:0:224} [start] ConnectionPool::addConnection m_oracleHome:/u01/11.2.0/grid m_oracleSid:+ASM2 m_usrOraEnv: pConnxn:
    01fcdf10
    2011-10-24 19:35:48.733: [ora.asm][9] {0:0:224} [start] Utils::getCrsHome crsHome /u01/11.2.0/grid
    2011-10-24 19:35:51.969: [ora.asm][14] {0:0:224} [check] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/11.2.0/grid/bin/oracle)(ARGV0=o
    racle+ASM2)(ENVS='ORACLE_HOME=/u01/11.2.0/grid,ORACLE_SID=+ASM2')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM2)))
    2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] ConnectionPool::getConnection 260 pConnxn 013e40a0
    2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] DgpAgent::getConnxn connected
    2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] InstConnection::connectInt: server not attached
    2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    SVR4 Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] InstConnection::connectInt (2) Exception OCIException
    2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] InstConnection:connect:excp OCIException OCI error 1034
    2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus excp ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    SVR4 Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0
    2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus asm inst is down or going down
    2011-10-24 19:35:52.191: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus dgName CRS ret 1
    2011-10-24 19:35:52.191: [ora.asm][14] {0:0:224} [check] (:CLSN00100:)DgpAgent::ocrDgCheck OCR dgName CRS state 1
    2011-10-24 19:35:52.192: [ora.asm][14] {0:0:224} [check] ConnectionPool::releaseConnection InstConnection 013e40a0
    2011-10-24 19:35:52.192: [ora.asm][14] {0:0:224} [check] AsmAgent::check ocrCheck 2 m_OcrOnline 0 m_OcrTimer 0
    2011-10-24 19:35:52.193: [ora.asm][14] {0:0:224} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
    2011-10-24 19:35:52.197: [ COMMCRS][23]clsc_connect: (1020d39d0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))
    Please advice for any workaround or a metalink note.
    Thanks in advance!

    Thanks for the fast reply!
    - Yes, the shared storage is accessible.
    - The alert log for the +ASM2 clearly shows that ASM instance has started normally using default parameters and at one point PMON process dumped.
    - The system logs just shows that there is an error executing "crswrapexece.pl"
    System Log
    ===================
    *Oct 24 19:25:03 host2 root: [ID 702911 user.error] exec /u01/11.2.0/grid/perl/bin/perl -I/u01/11.2.0/grid/perl/lib /u01/11.2.0/grid/bin/crswrapexece.pl /*
    u01/11.2.0/grid/crs/install/s_crsconfig_host2_env.txt /u01/11.2.0/grid/bin/ohasd.bin "reboot"
    Oct 24 19:26:33 host2 oracleoks: [ID 902884 kern.notice] [Oracle OKS] mallocing log buffer, size=10485760
    Oct 24 19:26:33 host2 oracleoks: [ID 714332 kern.notice] [Oracle OKS] log buffer = 0x301780fcb50, size 10485760
    Oct 24 19:26:33 host2 oracleoks: [ID 400061 kern.notice] NOTICE: [Oracle OKS] ODLM hash size 16384
    Oct 24 19:26:33 host2 oracleoks: [ID 160659 kern.notice] NOTICE: OKSK-00004: Module load succeeded. Build information: (LOW DEBUG) USM_11.2.0.3.0_SOLAR
    IS.SPARC64_110803.1 2011/08/11 02:38:30
    Oct 24 19:26:33 host2 pseudo: [ID 129642 kern.info] pseudo-device: oracleadvm0
    Oct 24 19:26:33 host2 genunix: [ID 936769 kern.info] oracleadvm0 is /pseudo/oracleadvm@0
    Oct 24 19:26:33 host2 oracleoks: [ID 141287 kern.notice] NOTICE: ADVMK-00001: Module load succeeded. Build information: (LOW DEBUG) - USM_11.2.0.3.0_SOL
    ARIS.SPARC64_110803.1 built on 2011/08/11 02:40:17.
    Oct 24 19:26:33 host2 oracleacfs: [ID 202941 kern.notice] NOTICE: [Oracle ACFS] FCB hash size 16384
    Oct 24 19:26:33 host2 oracleacfs: [ID 671725 kern.notice] NOTICE: [Oracle ACFS] buffer cache size 511MB (79884 buckets)
    Oct 24 19:26:33 host2 oracleacfs: [ID 730054 kern.notice] NOTICE: [Oracle ACFS] DLM hash size 16384
    Oct 24 19:26:33 host2 oracleoks: [ID 617314 kern.notice] NOTICE: ACFSK-0037: Module load succeeded. Build information: (LOW DEBUG) USM_11.2.0.3.0_SOLAR
    IS.SPARC64_110803.1 2011/08/11 02:42:45
    Oct 24 19:26:33 host2 pseudo: [ID 129642 kern.info] pseudo-device: oracleacfs0
    Oct 24 19:26:33 host2 genunix: [ID 936769 kern.info] oracleacfs0 is /pseudo/oracleacfs@0
    Oct 24 19:26:36 host2 oracleoks: [ID 621795 kern.notice] NOTICE: OKSK-00010: Persistent OKS log opened at /u01/11.2.0/grid/log/host2/acfs/acfs.log.0.
    Oct 24 19:31:37 host2 last message repeated 1 time
    Oct 24 19:33:05 host2 CLSD: [ID 770310 daemon.notice] The clock on host host2 has been updated by the Cluster Time Synchronization Service to be synchr
    onous with the mean cluster time.
    ASM alert log
    ====================================================================
    <msg time='2011-10-24T19:35:48.776+01:00' org_id='oracle' comp_id='asm'
    client_id='' type='UNKNOWN' level='16'
    host_id='host2' host_addr='10.172.16.200' module=''
    pid='26406'>
    <txt>System state dump requested by (instance=2, osid=26396 (PMON)), summary=[abnormal instance termination].
    </txt>
    </msg>
    <msg time='2011-10-24T19:35:48.778+01:00' org_id='oracle' comp_id='asm'
    client_id='' type='UNKNOWN' level='16'
    host_id='host2' host_addr='10.172.16.200' module=''
    pid='26406'>
    <txt>System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_26406.trc
    </txt>
    </msg>
    <msg time='2011-10-24T19:35:48.927+01:00' org_id='oracle' comp_id='asm'
    type='UNKNOWN' level='16' host_id='host2'
    host_addr='10.172.16.200' pid='26470'>
    <txt>ORA-1092 : opitsk aborting process
    </txt>
    </msg>
    <msg time='2011-10-24T19:35:49.128+01:00' org_id='oracle' comp_id='asm'
    type='UNKNOWN' level='16' host_id='host2'
    host_addr='10.172.16.200' pid='26472'>
    <txt>ORA-1092 : opitsk aborting process
    </txt>
    </msg>
    Output from "/u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_26406.trc"
    REQUEST:system state dump at level 10, requested by (instance=2, osid=26396 (PMON)), summary=[abnormal instance termination].
    kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).
    ===================================================
    SYSTEM STATE (level=10)
    Orapids on dead process list: [count = 0]
    PROCESS 1:
    SO: 0x3df098b50, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df098b50, name=process, file=ksu.h LINE:12616 ID:, pg=0
    (process) Oracle pid:1, ser:0, calls cur/top: 0x0/0x0
    flags : (0x20) PSEUDO
    flags2: (0x0), flags3: (0x10)
    intr error: 0, call error: 0, sess error: 0, txn error 0
    intr queue: empty
    ksudlp FALSE at location: 0
    (post info) last post received: 0 0 0
    last post received-location: No post
    last process to post me: none
    last post sent: 0 0 0
    last post sent-location: No post
    last process posted by me: none
    (latch info) wait_event=0 bits=0
    O/S info: user: , term: , ospid: (DEAD)
    OSD pid info: Unix process pid: 0, image: PSEUDO
    SO: 0x38000cef0, type: 5, owner: 0x3df098b50, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x0, name=kss parent, file=kss2.h LINE:138 ID:, pg=0
    PSO child state object changes :
    Dump of memory from 0x00000003DF722AC0 to 0x00000003DF722CC8
    3DF722AC0 00000000 00000000 00000000 00000000 [................]
    Repeat 31 times
    3DF722CC0 00000000 00000000 [........]
    PROCESS 2: PMON
    SO: 0x3df099bf8, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df099bf8, name=process, file=ksu.h LINE:12616 ID:, pg=0
    (process) Oracle pid:2, ser:1, calls cur/top: 0x3db6c8d30/0x3db6c8d30
    flags : (0xe) SYSTEM
    flags2: (0x0), flags3: (0x10)
    intr error: 0, call error: 0, sess error: 0, txn error 0
    intr queue: empty
    ksudlp FALSE at location: 0
    (post info) last post received: 0 0 136
    last post received-location: kjm.h LINE:1228 ID:kjmdmi: pmon to attach
    last process to post me: 3df0a2138 1 6
    last post sent: 0 0 137
    last post sent-location: kjm.h LINE:1230 ID:kjiath: pmon attached
    last process posted by me: 3df0a2138 1 6
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: 0x3debbbf40
    O/S info: user: grid, term: UNKNOWN, ospid: 26396
    OSD pid info: Unix process pid: 26396, image: oracle@host2 (PMON)
    SO: 0x3d8800c18, type: 30, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df099bf8, name=ges process, file=kji.h LINE:3669 ID:, pg=0
    GES MSG BUFFERS: st=emp chunk=0x0 hdr=0x0 lnk=0x0 flags=0x0 inc=0
    outq=0 sndq=0 opid=0 prmb=0x0
    mbg=(0 0) mbg=(0 0) mbg[r]=(0 0)
    fmq=(0 0) fmq=(0 0) fmq[r]=(0 0)
    mop[s]=0 mop[q]=0 pendq=0 zmbq=0
    nonksxp_recvs=0
    ------------process 3d8800c18--------------------
    proc version : 0
    Local inst : 2
    pid : 26396
    lkp_inst : 2
    svr_mode : 0
    proc state : KJP_FROZEN
    Last drm hb acked : 0
    flags : x50
    ast_rcvd_svrmod : 0
    current lock op : 0
    Total accesses : 1
    Imm. accesses : 0
    Locks on ASTQ : 0
    Locks Pending AST : 0
    Granted locks : 0
    AST_Q:
    PENDING_Q:
    GRANTED_Q:
    SO: 0x3d9835198, type: 14, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
    proc=0x3df099bf8, name=channel handle, file=ksr2.h LINE:367 ID:, pg=0
    (broadcast handle) 3d9835198 flag: (2) ACTIVE SUBSCRIBER,
    owner: 3df099bf8 - ospid: 26396
    event: 1, last message event: 1,
    last message waited event: 1,
    next message: 0(0), messages read: 0
    channel: (3d9934df8) PMON actions channel [name: 2]
    scope: 7, event: 1, last mesage event: 0,
    publishers/subscribers: 0/1,
    messages published: 0
    heuristic msg queue length: 0
    SO: 0x3d9835008, type: 14, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
    proc=0x3df099bf8, name=channel handle, file=ksr2.h LINE:367 ID:, pg=0
    (broadcast handle) 3d9835008 flag: (2) ACTIVE SUBSCRIBER,
    owner: 3df099bf8 - ospid: 26396
    event: 1, last message event: 1,
    last message waited event: 1,
    next message: 0(0), messages read: 0
    channel: (3d9941e40) scumnt mount lock [name: 157]
    scope: 1, event: 12, last mesage event: 0,
    publishers/subscribers: 0/12,
    messages published: 0
    heuristic msg queue length: 0
    SO: 0x3de4a2b80, type: 4, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df099bf8, name=session, file=ksu.h LINE:12624 ID:, pg=0
    (session) sid: 33 ser: 1 trans: 0x0, creator: 0x3df099bf8
    flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
    flags2: (0x409) -/-/INC
    DID: , short-term DID:
    txn branch: 0x0
    oct: 0, prv: 0, sql: 0x0, psql: 0x0, user: 0/SYS
    ksuxds FALSE at location: 0
    service name: SYS$BACKGROUND
    Current Wait Stack:
    Not in wait; last wait ended 0.666415 sec ago
    Wait State:
    fixed_waits=0 flags=0x21 boundary=0x0/-1
    Session Wait History:
    elapsed time of 0.666593 sec since last wait
    0: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=63 seq_num=64 snap_id=1
    wait times: snap=3.000089 sec, exc=3.000089 sec, total=3.000089 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.002067 sec of elapsed time
    1: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=62 seq_num=63 snap_id=1
    wait times: snap=3.010111 sec, exc=3.010111 sec, total=3.010111 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.001926 sec of elapsed time
    2: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=61 seq_num=62 snap_id=1
    wait times: snap=3.125286 sec, exc=3.125286 sec, total=3.125286 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.003361 sec of elapsed time
    3: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=60 seq_num=61 snap_id=1
    wait times: snap=3.000081 sec, exc=3.000081 sec, total=3.000081 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.002102 sec of elapsed time
    4: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0

  • Root.sh fails - asm won't shut down

    I am trying to instal clusterware 11gR2 on Oracle Enterprise Linux 5. This is all running an Oracle Virtualbox environment, using ASM for the cluster disk in an openfiler. 32 bit installation, 1GB memory allocated to each node environment. I had to ignore errors for the following:
    < 1.5 GB memory
    small swap file
    ncsd and ntpd processes not running (not connected to internet), I understand Oracle will install it's own timer service if this is not present
    Running root.sh fails because asm won't shut down properly:
    [root@odbn1 grid]# ./root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /u01/app/grid
    Enter the full pathname of the local bin directory: [usr/local/bin]:
    Copying dbhome to /usr/local/bin ...
    Copying oraenv to /usr/local/bin ...
    Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-08-24 14:46:44: Parsing the host name
    2010-08-24 14:46:44: Checking for super user privileges
    2010-08-24 14:46:44: User has super user privileges
    Using configuration parameter file: /u01/app/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    root wallet
    root wallet cert
    root cert export
    peer wallet
    profile reader wallet
    pa wallet
    peer wallet keys
    pa wallet keys
    peer cert request
    pa cert request
    peer cert
    pa cert
    peer root cert TP
    profile reader root cert TP
    pa root cert TP
    peer pa cert TP
    pa peer cert TP
    profile reader pa cert TP
    profile reader peer cert TP
    peer user cert
    pa user cert
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-2672: Attempting to start 'ora.gipcd' on 'odbn1'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'odbn1'
    CRS-2676: Start of 'ora.mdnsd' on 'odbn1' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'odbn1' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'odbn1'
    CRS-2676: Start of 'ora.gpnpd' on 'odbn1' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'odbn1'
    CRS-2676: Start of 'ora.cssdmonitor' on 'odbn1' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'odbn1'
    CRS-2672: Attempting to start 'ora.diskmon' on 'odbn1'
    CRS-2676: Start of 'ora.diskmon' on 'odbn1' succeeded
    CRS-2676: Start of 'ora.cssd' on 'odbn1' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'odbn1'
    CRS-2676: Start of 'ora.ctssd' on 'odbn1' succeeded
    ASM created and started successfully.
    DiskGroup CLSVOL1 created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    CRS-2672: Attempting to start 'ora.crsd' on 'odbn1'
    CRS-2676: Start of 'ora.crsd' on 'odbn1' succeeded
    CRS-4256: Updating the profile
    Successful addition of voting disk 502348953fc24f1cbf9c9f0fdf5cf2e0.
    Successfully replaced voting disk group with +CLSVOL1.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ## STATE File Universal Id File Name Disk group
    1. ONLINE 502348953fc24f1cbf9c9f0fdf5cf2e0 (/dev/oracleasm/disks/CRS) [CLSVOL1]
    Located 1 voting disk(s).
    CRS-2673: Attempting to stop 'ora.crsd' on 'odbn1'
    CRS-2677: Stop of 'ora.crsd' on 'odbn1' succeeded
    CRS-2673: Attempting to stop 'ora.asm' on 'odbn1'
    ORA-15097: cannot SHUTDOWN ASM instance with connected client
    CRS-2675: Stop of 'ora.asm' on 'odbn1' failed
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/app/grid/bin/crsctl stop resource ora.asm -init
    Stop of resource "ora.asm -init" failed
    Failed to stop ASM
    CRS-2673: Attempting to stop 'ora.ctssd' on 'odbn1'
    CRS-2677: Stop of 'ora.ctssd' on 'odbn1' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'odbn1'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'odbn1' succeeded
    CRS-2529: Unable to act on 'ora.cssd' because that would require stopping or relocating 'ora.asm', but the force option was not specified
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/app/grid/bin/crsctl stop resource ora.cssd -init
    Failed to exit exclusive mode
    Initial cluster configuration failed. See /u01/app/grid/cfgtoollogs/crsconfig/rootcrs_odbn1.log for details
    [root@odbn1 grid]#
    The rootcrs_odbn1.log shows the same error:
    [root@odbn1 crsconfig]# tail -50 rootcrs_odbn1.log
    2010-08-24 15:10:38: /bin/su successfully executed
    2010-08-24 15:10:38: /u01/app/grid/gpnp/odbn1/wallets/prdr/cwallet.sso => /u01/app/grid/gpnp/wallets/prdr/cwallet.sso
    2010-08-24 15:10:38: rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/prdr/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/prdr/cwallet.sso -nodelist odbn1,odbn2
    2010-08-24 15:10:38: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/prdr/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/prdr/cwallet.sso -nodelist odbn1,odbn2
    2010-08-24 15:10:38: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/prdr/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/prdr/cwallet.sso -nodelist odbn1,odbn2 '
    2010-08-24 15:11:20: Removing file /tmp/file1ye3x8
    2010-08-24 15:11:21: Successfully removed file: /tmp/file1ye3x8
    2010-08-24 15:11:21: /bin/su successfully executed
    2010-08-24 15:11:21: /u01/app/grid/gpnp/odbn1/wallets/pa/cwallet.sso => /u01/app/grid/gpnp/wallets/pa/cwallet.sso
    2010-08-24 15:11:21: rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/pa/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/pa/cwallet.sso -nodelist odbn1,odbn2
    2010-08-24 15:11:21: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/pa/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/pa/cwallet.sso -nodelist odbn1,odbn2
    2010-08-24 15:11:21: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/pa/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/pa/cwallet.sso -nodelist odbn1,odbn2 '
    2010-08-24 15:11:49: Removing file /tmp/filelEb5Lp
    2010-08-24 15:11:49: Successfully removed file: /tmp/filelEb5Lp
    2010-08-24 15:11:50: /bin/su successfully executed
    2010-08-24 15:11:50: /u01/app/grid/gpnp/odbn1/wallets/root/b64certificate.txt => /u01/app/grid/gpnp/wallets/root/b64certificate.txt
    2010-08-24 15:11:50: rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/root/b64certificate.txt -destfile /u01/app/grid/gpnp/wallets/root/b64certificate.txt -nodelist odbn1,odbn2
    2010-08-24 15:11:50: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/root/b64certificate.txt -destfile /u01/app/grid/gpnp/wallets/root/b64certificate.txt -nodelist odbn1,odbn2
    2010-08-24 15:11:50: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/root/b64certificate.txt -destfile /u01/app/grid/gpnp/wallets/root/b64certificate.txt -nodelist odbn1,odbn2 '
    2010-08-24 15:12:26: Removing file /tmp/fileUQzFxE
    2010-08-24 15:12:27: Successfully removed file: /tmp/fileUQzFxE
    2010-08-24 15:12:27: /bin/su successfully executed
    2010-08-24 15:12:27: /u01/app/grid/gpnp/odbn1/wallets/peer/cert.txt => /u01/app/grid/gpnp/wallets/peer/cert.txt
    2010-08-24 15:12:27: rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/peer/cert.txt -destfile /u01/app/grid/gpnp/wallets/peer/cert.txt -nodelist odbn1,odbn2
    2010-08-24 15:12:27: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/peer/cert.txt -destfile /u01/app/grid/gpnp/wallets/peer/cert.txt -nodelist odbn1,odbn2
    2010-08-24 15:12:27: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/peer/cert.txt -destfile /u01/app/grid/gpnp/wallets/peer/cert.txt -nodelist odbn1,odbn2 '
    2010-08-24 15:12:47: Removing file /tmp/filevnw3D8
    2010-08-24 15:12:47: Successfully removed file: /tmp/filevnw3D8
    2010-08-24 15:12:47: /bin/su successfully executed
    2010-08-24 15:12:47: /u01/app/grid/gpnp/odbn1/wallets/pa/cert.txt => /u01/app/grid/gpnp/wallets/pa/cert.txt
    2010-08-24 15:12:47: rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/pa/cert.txt -destfile /u01/app/grid/gpnp/wallets/pa/cert.txt -nodelist odbn1,odbn2
    2010-08-24 15:12:47: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/pa/cert.txt -destfile /u01/app/grid/gpnp/wallets/pa/cert.txt -nodelist odbn1,odbn2
    2010-08-24 15:12:47: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/odbn1/wallets/pa/cert.txt -destfile /u01/app/grid/gpnp/wallets/pa/cert.txt -nodelist odbn1,odbn2 '
    2010-08-24 15:13:19: Removing file /tmp/fileArkUFi
    2010-08-24 15:13:20: Successfully removed file: /tmp/fileArkUFi
    2010-08-24 15:13:20: /bin/su successfully executed
    2010-08-24 15:13:20: Exiting exclusive mode
    2010-08-24 15:13:41: Command return code of 1 (256) from command: /u01/app/grid/bin/crsctl stop resource ora.asm -init
    2010-08-24 15:13:41: Stop of resource "ora.asm -init" failed
    2010-08-24 15:13:41: Failed to stop ASM
    2010-08-24 15:14:44: Command return code of 1 (256) from command: /u01/app/grid/bin/crsctl stop resource ora.cssd -init
    2010-08-24 15:14:44: CSS shutdown failed
    2010-08-24 15:14:44: Failed to exit exclusive mode
    2010-08-24 15:14:44: Initial cluster configuration failed. See /u01/app/grid/cfgtoollogs/crsconfig/rootcrs_odbn1.log for details
    [root@odbn1 crsconfig]#
    Has anyone seen this before? All help greatly appreciated!

    Hi Sebastian,
    Thank you for your quick reply. Here's what I tried:
    1. reset the install as you suggested, and tried running root.sh again. The exact same thing happened.
    2. restored to an openfiler snapshots that were taken right before the failed upgrade. Increased memory on both VMs to 1280M as you suggested. Rebooted the virtual environment and ran the installation again. This time, the root.sh script froze my environment - not enough memory to support this.
    3. restored the openfiler snapshots again. Increased memory on one node (odbn1) to 1500M. Ran the install for a single node cluster. It got past this error.
    Thank you!!!
    Also, thanks for the tip on the ntp.init file. That did eliminate the error for ntpd not running during the installation.

  • 11G R2 root.sh failed on first node with OLE fetch parameter error

    I have successfully installed 11G R2.1 on Centos 5.4 64 bit.
    Now it's coming to install 11G R2.2 on Redhat 5.4 64bit with HDS storrage.
    [grid@dmdb1 grid]$ uname -a
    Linux dmdb1 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
    I passed all pre-ins requirements except shared storage. However, I manually verify it with no problems.
    [grid@dmdb1 grid]$ ./runcluvfy.sh stage -pre crsinst -fixup -n dmdb1,dmdb2,dmdb3,dmdb4 -verbose|grep -i fail
    [grid@dmdb1 grid]$ ./runcluvfy.sh stage -post hwos -n dmdb1,dmdb2,dmdb3,dmdb4 -verbose|grep -i fail
    [grid@dmdb1 grid]$ ./runcluvfy.sh comp sys -n dmdb1,dmdb2,dmdb3,dmdb4 -p crs -osdba dba -orainv oinstall
    Verifying system requirement
    Total memory check passed
    Available memory check passed
    Swap space check passed
    Free disk space check passed for "dmdb4:/tmp"
    Free disk space check passed for "dmdb3:/tmp"
    Free disk space check passed for "dmdb2:/tmp"
    Free disk space check passed for "dmdb1:/tmp"
    User existence check passed for "grid"
    Group existence check passed for "oinstall"
    Group existence check passed for "dba"
    Membership check for user "grid" in group "oinstall" [as Primary] passed
    Membership check for user "grid" in group "dba" passed
    Run level check passed
    Hard limits check passed for "maximum open file descriptors"
    Soft limits check passed for "maximum open file descriptors"
    Hard limits check passed for "maximum user processes"
    Soft limits check passed for "maximum user processes"
    System architecture check passed
    Kernel version check passed
    Kernel parameter check passed for "semmsl"
    Kernel parameter check passed for "semmns"
    Kernel parameter check passed for "semopm"
    Kernel parameter check passed for "semmni"
    Kernel parameter check passed for "shmmax"
    Kernel parameter check passed for "shmmni"
    Kernel parameter check passed for "shmall"
    Kernel parameter check passed for "file-max"
    Kernel parameter check passed for "ip_local_port_range"
    Kernel parameter check passed for "rmem_default"
    Kernel parameter check passed for "rmem_max"
    Kernel parameter check passed for "wmem_default"
    Kernel parameter check passed for "wmem_max"
    Kernel parameter check passed for "aio-max-nr"
    Package existence check passed for "make-3.81"
    Package existence check passed for "binutils-2.17.50.0.6"
    Package existence check passed for "gcc-4.1"
    Package existence check passed for "libaio-0.3.106 (i386)"
    Package existence check passed for "libaio-0.3.106 (x86_64)"
    Package existence check passed for "glibc-2.5-24 (i686)"
    Package existence check passed for "glibc-2.5-24 (x86_64)"
    Package existence check passed for "compat-libstdc++-33-3.2.3 (i386)"
    Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)"
    Package existence check passed for "elfutils-libelf-0.125 (x86_64)"
    Package existence check passed for "elfutils-libelf-devel-0.125"
    Package existence check passed for "glibc-common-2.5"
    Package existence check passed for "glibc-devel-2.5 (i386)"
    Package existence check passed for "glibc-devel-2.5 (x86_64)"
    Package existence check passed for "glibc-headers-2.5"
    Package existence check passed for "gcc-c++-4.1.2"
    Package existence check passed for "libaio-devel-0.3.106 (i386)"
    Package existence check passed for "libaio-devel-0.3.106 (x86_64)"
    Package existence check passed for "libgcc-4.1.2 (i386)"
    Package existence check passed for "libgcc-4.1.2 (x86_64)"
    Package existence check passed for "libstdc++-4.1.2 (i386)"
    Package existence check passed for "libstdc++-4.1.2 (x86_64)"
    Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)"
    Package existence check passed for "sysstat-7.0.2"
    Package existence check passed for "unixODBC-2.2.11 (i386)"
    Package existence check passed for "unixODBC-2.2.11 (x86_64)"
    Package existence check passed for "unixODBC-devel-2.2.11 (i386)"
    Package existence check passed for "unixODBC-devel-2.2.11 (x86_64)"
    Package existence check passed for "ksh-20060214"
    Check for multiple users with UID value 0 passed
    Verification of system requirement was successful.
    [grid@dmdb1 grid]$ ./runcluvfy.sh comp sys -n dmdb1,dmdb2,dmdb3,dmdb4 -p database -osdba dba -orainv oinstall|grep -i fail
    [grid@dmdb1 grid]$ ./runcluvfy.sh comp ssa -n dmdb1,dmdb2,dmdb3,dmdb4
    Verifying shared storage accessibility
    Checking shared storage accessibility...
    Storage operation failed
    Shared storage check failed on nodes "dmdb4,dmdb3,dmdb2,dmdb1"
    Verification of shared storage accessibility was unsuccessful on all the specified nodes.
    I followed below article to verify shared storage issues:
    http://www.webofwood.com/rac/oracle-response-to-shared-storage-check-failed-on-nodes/
    it's ok.
    So I skipped SSA issue and go on install with (./runInstaller -ignoreInternalDriverError).
    However, when I ran root.sh with below error:
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'dmdb1'
    CRS-2677: Stop of 'ora.mdnsd' on 'dmdb1' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'dmdb1'
    CRS-2677: Stop of 'ora.gipcd' on 'dmdb1' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    CRS-2672: Attempting to start 'ora.gipcd' on 'dmdb1'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'dmdb1'
    CRS-2676: Start of 'ora.gipcd' on 'dmdb1' succeeded
    CRS-2676: Start of 'ora.mdnsd' on 'dmdb1' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'dmdb1'
    CRS-2676: Start of 'ora.gpnpd' on 'dmdb1' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dmdb1'
    CRS-2676: Start of 'ora.cssdmonitor' on 'dmdb1' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'dmdb1'
    CRS-2672: Attempting to start 'ora.diskmon' on 'dmdb1'
    CRS-2676: Start of 'ora.diskmon' on 'dmdb1' succeeded
    CRS-2674: Start of 'ora.cssd' on 'dmdb1' failed
    CRS-2679: Attempting to clean 'ora.cssd' on 'dmdb1'
    CRS-2681: Clean of 'ora.cssd' on 'dmdb1' succeeded
    CRS-2673: Attempting to stop 'ora.diskmon' on 'dmdb1'
    CRS-2677: Stop of 'ora.diskmon' on 'dmdb1' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'dmdb1'
    CRS-2677: Stop of 'ora.gpnpd' on 'dmdb1' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'dmdb1'
    CRS-2677: Stop of 'ora.mdnsd' on 'dmdb1' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'dmdb1'
    CRS-2677: Stop of 'ora.gipcd' on 'dmdb1' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /opt/app/11.2.0/grid/bin/crsctl start resource ora.ctssd -init
    Start of resource "ora.ctssd -init" failed
    Clusterware exclusive mode start of resource ora.ctssd failed
    CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /opt/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
    Stop of resource "ora.crsd -init" failed
    Failed to stop CRSD
    CRS-2500: Cannot stop resource 'ora.asm' as it is not running
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /opt/app/11.2.0/grid/bin/crsctl stop resource ora.asm -init
    Stop of resource "ora.asm -init" failed
    Failed to stop ASM
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'dmdb1'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'dmdb1' succeeded
    Initial cluster configuration failed. See /opt/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_dmdb1.log for details
    I manually ran '/opt/app/11.2.0/grid/bin/crsctl start resource ora.ctssd -init' and got below erros from /opt/app/11.2.0/grid/log/dmdb1/cssd/ocssd.log
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2011-09-23 19:06:41.501: [    CSSD][1812336384]clssscmain: Starting CSS daemon, version 11.2.0.1.0, in (exclusive) mode with uniqueness value 1316776001
    2011-09-23 19:06:41.502: [    CSSD][1812336384]clssscmain: Environment is production
    2011-09-23 19:06:41.502: [    CSSD][1812336384]clssscmain: Core file size limit extended
    2011-09-23 19:06:41.515: [    CSSD][1812336384]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
    2011-09-23 19:06:41.515: [    CSSD][1812336384]clssscSetPrivEnv: IPMI device not installed on this node
    2011-09-23 19:06:41.517: [    CSSD][1812336384]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
    2011-09-23 19:06:41.539: [    CSSD][1812336384]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
    2011-09-23 19:06:41.539: [    CSSD][1812336384]clssscExtendLimits: The current soft limit for locked memory is 4294967295, hard limit is 4294967295
    2011-09-23 19:06:41.541: [    CSSD][1812336384]clssscmain: Running as user grid
    anybody can help me fix it?

    I opened on SR for this case.
    it's ok now.
    Below is from Oracle Global Service request:
    === ODM Action Plan ===
    Dear customer, after went through the uploaded log files, we found the issue looks like
    bug 9732641 : The clusterware gpnpd process crashes when there is more than 1 cluster with the same name.
    To narrow down the issue, pls apply the following steps.
    1. Pls clean the previous configuration with below steps, then run root.sh script on node1 again.
    1.1 remove current configuration.
    $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force
    1.2 remove other related files.
    if $GI_BASE/Clusterware/ckptGridHA_.xml still there, please remove it manually with "rm" command on all nodes
    If the gpnp profile is still there, pls clean up them, then rebuild require directories.
    $ rm -rf $GRID_HOME/gpnp/*
    $ mkdir -p $GRID_HOME/gpnp/profiles/peer $GRID_HOME/gpnp/wallets/peer $GRID_HOME/gpnp/wallets/prdr $GRID_HOME/gpnp/wallets/pa $GRID_HOME/gpnp/wallets/root
    2. After the previous configuration was cleaned up, pls rerun the root.sh script again. If the issue still there, pls upload the following:
    Everything under <GI_HOME>/log
    Everything under <ORACLE_BAES for grid user>/cfgtoollogs
    Everything under <GI_HOME>/cfgtolllogs/crsconfig
    OS log(/var/log/messages)
    3. Pls also make sure there is only one GI running on your cluster.
    See /opt/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_dmdb1.log for details

  • Root.sh fails for 11gR2 Grid Infrastructure installation on AIX 6.1

    Hello all,
    root.sh fails with the errors below. SR with Oracle opened. Will post the resolution when it is available. Any insights in the meantime? Thank you!
    System information:
    OS: AIX 6.1
    Runcluvfy.sh reported no issue
    Permissions on the raw devices set to 660 and ownership is oracle:dba
    Using external redundancy for ASM, ASM instance is online
    Permissions on block and raw device files
    system1:ux460p1> ls -l /dev/hdisk32
    brw-rw---- 1 oracle dba 17, 32 Mar 11 16:50 /dev/hdisk32
    system11:ux460p1> ls -l /dev/rhdisk32
    crw-rw---- 1 oracle dba 17, 32 Mar 12 15:52 /dev/rhdisk32
    ocrconfig.log
    racle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2010-03-15 19:17:19.773: [ OCRCONF][1]ocrconfig starts...
    2010-03-15 19:17:19.775: [ OCRCONF][1]Upgrading OCR data
    2010-03-15 19:17:20.474: [  OCRASM][1]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
    2010-03-15 19:17:20.474: [  OCRASM][1]proprasmo: Error in open/create file in dg [DATA]
    [  OCRASM][1]SLOS : [clsuSlosFormatDiag called with non-error slos.]
    2010-03-15 19:17:20.603: [  OCRRAW][1]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
    2010-03-15 19:17:20.603: [  OCRRAW][1]proprioo: No OCR/OLR devices are usable
    2010-03-15 19:17:20.603: [  OCRASM][1]proprasmcl: asmhandle is NULL
    2010-03-15 19:17:20.603: [  OCRRAW][1]proprinit: Could not open raw device
    2010-03-15 19:17:20.603: [  OCRASM][1]proprasmcl: asmhandle is NULL
    2010-03-15 19:17:20.604: [ default][1]a_init:7!: Backend init unsuccessful : [26]
    2010-03-15 19:17:20.604: [ OCRCONF][1]Exporting OCR data to [OCRUPGRADEFILE]
    2010-03-15 19:17:20.604: [  OCRAPI][1]a_init:7!: Backend init unsuccessful : [33]
    2010-03-15 19:17:20.605: [ OCRCONF][1]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
    2010-03-15 19:17:20.841: [  OCRASM][1]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
    2010-03-15 19:17:20.841: [  OCRASM][1]proprasmo: Error in open/create file in dg [DATA]
    [  OCRASM][1]SLOS : [clsuSlosFormatDiag called with non-error slos.]
    2010-03-15 19:17:20.966: [  OCRRAW][1]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
    2010-03-15 19:17:20.966: [  OCRRAW][1]proprioo: No OCR/OLR devices are usable
    2010-03-15 19:17:20.966: [  OCRASM][1]proprasmcl: asmhandle is NULL
    2010-03-15 19:17:20.966: [  OCRRAW][1]proprinit: Could not open raw device
    2010-03-15 19:17:20.966: [  OCRASM][1]proprasmcl: asmhandle is NULL
    2010-03-15 19:17:20.966: [ default][1]a_init:7!: Backend init unsuccessful : [26]
    2010-03-15 19:17:21.412: [  OCRRAW][1]propriogid:1_2: INVALID FORMAT
    2010-03-15 19:17:21.412: [  OCRRAW][1]proprior: Header check from OCR device 0 offset 0 failed (26).
    2010-03-15 19:17:21.414: [  OCRRAW][1]ibctx: Failed to read the whole bootblock. Assumes invalid format.
    2010-03-15 19:17:21.414: [  OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22
    2010-03-15 19:17:21.534: [  OCRRAW][1]propriogid:1_2: INVALID FORMAT
    2010-03-15 19:17:21.701: [  OCRRAW][1]iniconfig:No 92 configuration
    2010-03-15 19:17:21.701: [  OCRAPI][1]a_init:6a: Backend init successful
    2010-03-15 19:17:21.764: [ OCRCONF][1]Initialized DATABASE keys
    2010-03-15 19:17:21.770: [ OCRCONF][1]Successfully set skgfr block 0
    2010-03-15 19:17:21.771: [ OCRCONF][1]Exiting [status=success]...
    **alert.log**
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2010-03-15 19:12:00.148
    [client(483478)]CRS-2106:The OLR location /u01/app/grid/cdata/ux460p1.olr is inaccessible. Details in /u01/app/grid/log/ux460p1/client/ocrconfig_483478.log.
    2010-03-15 19:12:00.171
    [client(483478)]CRS-2101:The OLR was formatted using version 3.
    2010-03-15 14:16:18.620
    [ohasd(471204)]CRS-2112:The OLR service started on node ux460p1.
    2010-03-15 14:16:18.720
    [ohasd(471204)]CRS-8017:location: /etc/oracle/lastgasp has 8 reboot advisory log files, 0 were announced and 0 errors occurred
    2010-03-15 14:16:18.847
    [ohasd(471204)]CRS-2772:Server 'ux460p1' has been assigned to pool 'Free'.
    2010-03-15 14:16:54.107
    [ctssd(340174)]CRS-2403:The Cluster Time Synchronization Service on host ux460p1 is in observer mode.
    2010-03-15 14:16:54.123
    [ctssd(340174)]CRS-2407:The new Cluster Time Synchronization Service reference node is host ux460p1.
    2010-03-15 14:16:54.917
    [ctssd(340174)]CRS-2401:The Cluster Time Synchronization Service started on host ux460p1.
    2010-03-15 19:17:21.414
    [client(376968)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/grid/log/ux460p1/client/ocrconfig_376968.log.
    2010-03-15 19:17:21.701
    [client(376968)]CRS-1001:The OCR was formatted using version 3.
    2010-03-15 14:17:24.888
    [crsd(303252)]CRS-1012:The OCR service started on node ux460p1.
    2010-03-15 14:17:56.344
    [ctssd(340174)]CRS-2405:The Cluster Time Synchronization Service on host ux460p1 is shutdown by user
    2010-03-15 14:19:14.855
    [ctssd(340188)]CRS-2403:The Cluster Time Synchronization Service on host ux460p1 is in observer mode.
    2010-03-15 14:19:14.870
    [ctssd(340188)]CRS-2407:The new Cluster Time Synchronization Service reference node is host ux460p1.
    2010-03-15 14:19:15.638
    [ctssd(340188)]CRS-2401:The Cluster Time Synchronization Service started on host ux460p1.
    2010-03-15 14:19:32.985
    [crsd(417946)]CRS-1012:The OCR service started on node ux460p1.
    2010-03-15 14:19:35.250
    [crsd(417946)]CRS-1201:CRSD started on node ux460p1.
    2010-03-15 14:19:35.698
    [ohasd(471204)]CRS-2765:Resource 'ora.crsd' has failed on server 'ux460p1'.
    2010-03-15 14:19:38.928

    Public and Private are on different devices and subnets.
    No logfile named: ocrconfig_7833.log
    I do have ocrconfig_7089.log and ocrconfig_8985.log
    Here is their contents:
    ocrconfig_7089.log:
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2010-11-09 13:38:32.518: [ OCRCONF][2819644944]ocrconfig starts...
    2010-11-09 13:38:32.542: [ OCRCONF][2819644944]Upgrading OCR data
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
    2010-11-09 13:38:32.576: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
    2010-11-09 13:38:32.576: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:38:32.576: [  OCRRAW][2819644944]proprioini: all disks are not OCR/OLR formatted
    2010-11-09 13:38:32.576: [  OCRRAW][2819644944]proprinit: Could not open raw device
    2010-11-09 13:38:32.576: [ default][2819644944]a_init:7!: Backend init unsuccessful : [26]
    2010-11-09 13:38:32.577: [ OCRCONF][2819644944]Exporting OCR data to [OCRUPGRADEFILE]
    2010-11-09 13:38:32.577: [  OCRAPI][2819644944]a_init:7!: Backend init unsuccessful : [33]
    2010-11-09 13:38:32.577: [ OCRCONF][2819644944]There was no previous version of OCR. error:[PROCL-33: Oracle Local Registry is not configured]
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
    2010-11-09 13:38:32.578: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
    2010-11-09 13:38:32.578: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:38:32.578: [  OCRRAW][2819644944]proprioini: all disks are not OCR/OLR formatted
    2010-11-09 13:38:32.578: [  OCRRAW][2819644944]proprinit: Could not open raw device
    2010-11-09 13:38:32.578: [ default][2819644944]a_init:7!: Backend init unsuccessful : [26]
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
    2010-11-09 13:38:32.579: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 0
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 1
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 2
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 3
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 4
    2010-11-09 13:38:32.579: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e54000 buflen 4096 retval 0 phy_offset 102400 retry 5
    2010-11-09 13:38:32.591: [  OCRRAW][2819644944]ibctx: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:38:32.591: [  OCRRAW][2819644944]proprinit:problem reading the bootblock or superbloc 22
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 0
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 1
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 2
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 3
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 4
    2010-11-09 13:38:32.591: [  OCROSD][2819644944]utread:3: Problem reading buffer 12e55000 buflen 4096 retval 0 phy_offset 102400 retry 5
    2010-11-09 13:38:32.591: [  OCRRAW][2819644944]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:38:32.681: [  OCRAPI][2819644944]a_init:6a: Backend init successful
    2010-11-09 13:38:32.699: [ OCRCONF][2819644944]Initialized DATABASE keys
    2010-11-09 13:38:32.700: [ OCRCONF][2819644944]Exiting [status=success]...
    ocrconfig_8985.log:
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2010-11-09 13:41:28.169: [ OCRCONF][2281741840]ocrconfig starts...
    2010-11-09 13:41:28.175: [ OCRCONF][2281741840]Upgrading OCR data
    2010-11-09 13:41:30.896: [  OCRASM][2281741840]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
    2010-11-09 13:41:30.896: [  OCRASM][2281741840]proprasmo: Error in open/create file in dg [DATA]
    [  OCRASM][2281741840]SLOS : [clsuSlosFormatDiag called with non-error slos.]
    2010-11-09 13:41:31.208: [  OCRRAW][2281741840]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
    2010-11-09 13:41:31.210: [  OCRRAW][2281741840]proprioo: No OCR/OLR devices are usable
    2010-11-09 13:41:31.210: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
    2010-11-09 13:41:31.210: [  OCRRAW][2281741840]proprinit: Could not open raw device
    2010-11-09 13:41:31.211: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
    2010-11-09 13:41:31.213: [ default][2281741840]a_init:7!: Backend init unsuccessful : [26]
    2010-11-09 13:41:31.214: [ OCRCONF][2281741840]Exporting OCR data to [OCRUPGRADEFILE]
    2010-11-09 13:41:31.216: [  OCRAPI][2281741840]a_init:7!: Backend init unsuccessful : [33]
    2010-11-09 13:41:31.216: [ OCRCONF][2281741840]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
    2010-11-09 13:41:32.214: [  OCRASM][2281741840]proprasmo: kgfoCheckMount return [0]. Cannot proceed with dirty open.
    2010-11-09 13:41:32.214: [  OCRASM][2281741840]proprasmo: Error in open/create file in dg [DATA]
    [  OCRASM][2281741840]SLOS : [clsuSlosFormatDiag called with non-error slos.]
    2010-11-09 13:41:32.535: [  OCRRAW][2281741840]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
    2010-11-09 13:41:32.535: [  OCRRAW][2281741840]proprioo: No OCR/OLR devices are usable
    2010-11-09 13:41:32.535: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
    2010-11-09 13:41:32.535: [  OCRRAW][2281741840]proprinit: Could not open raw device
    2010-11-09 13:41:32.535: [  OCRASM][2281741840]proprasmcl: asmhandle is NULL
    2010-11-09 13:41:32.536: [ default][2281741840]a_init:7!: Backend init unsuccessful : [26]
    2010-11-09 13:41:35.359: [  OCRRAW][2281741840]propriogid:1_2: INVALID FORMAT
    2010-11-09 13:41:35.361: [  OCRRAW][2281741840]proprior: Header check from OCR device 0 offset 0 failed (26).
    2010-11-09 13:41:35.363: [  OCRRAW][2281741840]ibctx: Failed to read the whole bootblock. Assumes invalid format.
    2010-11-09 13:41:35.363: [  OCRRAW][2281741840]proprinit:problem reading the bootblock or superbloc 22
    2010-11-09 13:41:35.843: [  OCRRAW][2281741840]propriogid:1_2: INVALID FORMAT
    2010-11-09 13:41:36.430: [  OCRRAW][2281741840]iniconfig:No 92 configuration
    2010-11-09 13:41:36.431: [  OCRAPI][2281741840]a_init:6a: Backend init successful
    2010-11-09 13:41:36.540: [ OCRCONF][2281741840]Initialized DATABASE keys
    2010-11-09 13:41:36.545: [ OCRCONF][2281741840]Successfully set skgfr block 0
    2010-11-09 13:41:36.552: [ OCRCONF][2281741840]Exiting [status=success]...
    Both of these log files show errors, then they show success??????

  • 11g R2 RAC - Grid Infrastructure installation - "root.sh" fails on node#2

    Hi there,
    I am trying to create a two node 11g R2 RAC on OEL 5.5 (32-bit) using VMWare virtual machines. I have correctly configured both nodes. Cluster Verification utility returns on following error \[which I believe can be ignored]:
    Checking daemon liveness...
    Liveness check failed for "ntpd"
    Check failed on nodes:
    rac2,rac1
    PRVF-5415 : Check to see if NTP daemon is running failed
    Clock synchronization check using Network Time Protocol(NTP) failed
    Pre-check for cluster services setup was unsuccessful on all the nodes.
    While Grid Infrastructure installation (for a Cluster option), things go very smooth until I run "root.sh" on node# 2. orainstRoot.sh ran OK on both. "root.sh" run OK on node# 1 and ends with:
    Checking swap space: must be greater than 500 MB.   Actual 1967 MB    Passed
    The inventory pointer is located at /etc/oraInst.loc
    The inventory is located at /u01/app/oraInventory
    *'UpdateNodeList' was successful.*
    *[root@rac1 ~]#*
    "root.sh" fails on rac2 (2nd node) with following error:
    CRS-2672: Attempting to start 'ora.evmd' on 'rac2'
    CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded
    Timed out waiting for the CRS stack to start.
    *[root@rac2 ~]#*
    I know this info may not be enough to figure out what the problem may be. Please let me know what should I look for to find the issue and fix it. Its been like almost two weeks now :-(
    Regards
    Amer

    Hi Zheng,
    ocssd.log is HUGE. So I am putting few of the last lines in the log file hoping they may give some clue:
    2011-07-04 19:49:24.007: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 2180 > margin 1500  cur_ms 36118424 lastalive 36116244
    2011-07-04 19:49:26.005: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 4150 > margin 1500 cur_ms 36120424 lastalive 36116274
    2011-07-04 19:49:26.006: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 4180 > margin 1500  cur_ms 36120424 lastalive 36116244
    2011-07-04 19:49:27.997: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:49:27.997: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:49:33.001: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:49:33.001: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:49:37.996: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:49:37.996: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:49:43.000: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:49:43.000: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:49:48.004: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:49:48.005: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:12.003: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:12.008: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1660 > margin 1500 cur_ms 36166424 lastalive 36164764
    2011-07-04 19:50:12.009: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1660 > margin 1500  cur_ms 36166424 lastalive 36164764
    2011-07-04 19:50:15.796: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 2130 > margin 1500  cur_ms 36170214 lastalive 36168084
    2011-07-04 19:50:16.996: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:50:16.996: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:17.826: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1540 > margin 1500 cur_ms 36172244 lastalive 36170704
    2011-07-04 19:50:17.826: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1570 > margin 1500  cur_ms 36172244 lastalive 36170674
    2011-07-04 19:50:21.999: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:50:21.999: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:26.011: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1740 > margin 1500 cur_ms 36180424 lastalive 36178684
    2011-07-04 19:50:26.011: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1620 > margin 1500  cur_ms 36180424 lastalive 36178804
    2011-07-04 19:50:27.004: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:50:27.004: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:28.002: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1700 > margin 1500 cur_ms 36182414 lastalive 36180714
    2011-07-04 19:50:28.002: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1790 > margin 1500  cur_ms 36182414 lastalive 36180624
    2011-07-04 19:50:31.998: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:50:31.998: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:37.001: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:50:37.002: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    *<end of log file>*And the alertrac2.log contains:
    *[root@rac2 rac2]# cat alertrac2.log*
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2011-07-02 16:43:51.571
    [client(16134)]CRS-2106:The OLR location /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olr is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/client/ocrconfig_16134.log.
    2011-07-02 16:43:57.125
    [client(16134)]CRS-2101:The OLR was formatted using version 3.
    2011-07-02 16:44:43.214
    [ohasd(16188)]CRS-2112:The OLR service started on node rac2.
    2011-07-02 16:45:06.446
    [ohasd(16188)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
    2011-07-02 16:53:30.061
    [ohasd(16188)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2011-07-02 16:53:55.042
    [cssd(17674)]CRS-1713:CSSD daemon is started in exclusive mode
    2011-07-02 16:54:38.334
    [cssd(17674)]CRS-1707:Lease acquisition for node rac2 number 2 completed
    [cssd(17674)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1 and is terminating; details at (:CSSNM00006:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log
    2011-07-02 16:54:38.464
    [cssd(17674)]CRS-1603:CSSD on node rac2 shutdown by user.
    2011-07-02 16:54:39.174
    [ohasd(16188)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac2'.
    2011-07-02 16:55:43.430
    [cssd(17945)]CRS-1713:CSSD daemon is started in clustered mode
    2011-07-02 16:56:02.852
    [cssd(17945)]CRS-1707:Lease acquisition for node rac2 number 2 completed
    2011-07-02 16:56:04.061
    [cssd(17945)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
    2011-07-02 16:56:18.350
    [cssd(17945)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
    2011-07-02 16:56:29.283
    [ctssd(18020)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
    2011-07-02 16:56:29.551
    [ctssd(18020)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
    2011-07-02 16:56:29.615
    [ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 16:56:29.616
    [ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 16:56:29.641
    [ctssd(18020)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
    [client(18052)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(18056)]CRS-10001:ACFS-9322: done.
    2011-07-02 17:01:40.963
    [ohasd(16188)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ohasd/ohasd.log.
    [client(18590)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(18594)]CRS-10001:ACFS-9322: done.
    2011-07-02 17:27:46.385
    [ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 17:27:46.385
    [ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 17:46:48.717
    [crsd(22519)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:46:49.641
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:46:51.459
    [crsd(22553)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:46:51.776
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:46:53.928
    [crsd(22574)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:46:53.956
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:46:55.834
    [crsd(22592)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:46:56.273
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:46:57.762
    [crsd(22610)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:46:58.631
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:00.259
    [crsd(22628)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:00.968
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:02.513
    [crsd(22645)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:03.309
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:05.081
    [crsd(22663)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:05.770
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:07.796
    [crsd(22681)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:08.257
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:10.733
    [crsd(22699)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:11.739
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:13.547
    [crsd(22732)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:14.111
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:14.112
    [ohasd(16188)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    2011-07-02 17:58:18.459
    [ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 17:58:18.459
    [ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    [client(26883)]CRS-10001:ACFS-9200: Supported
    2011-07-02 18:13:34.627
    [ctssd(18020)]CRS-2405:The Cluster Time Synchronization Service on host rac2 is shutdown by user
    2011-07-02 18:13:42.368
    [cssd(17945)]CRS-1603:CSSD on node rac2 shutdown by user.
    2011-07-02 18:15:13.877
    [client(27222)]CRS-2106:The OLR location /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olr is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/client/ocrconfig_27222.log.
    2011-07-02 18:15:14.011
    [client(27222)]CRS-2101:The OLR was formatted using version 3.
    2011-07-02 18:15:23.226
    [ohasd(27261)]CRS-2112:The OLR service started on node rac2.
    2011-07-02 18:15:23.688
    [ohasd(27261)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
    2011-07-02 18:15:24.064
    [ohasd(27261)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
    2011-07-02 18:16:29.761
    [ohasd(27261)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2011-07-02 18:16:30.190
    [gpnpd(28498)]CRS-2328:GPNPD started on node rac2.
    2011-07-02 18:16:41.561
    [cssd(28562)]CRS-1713:CSSD daemon is started in exclusive mode
    2011-07-02 18:16:49.111
    [cssd(28562)]CRS-1707:Lease acquisition for node rac2 number 2 completed
    2011-07-02 18:16:49.166
    [cssd(28562)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
    [cssd(28562)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1 and is terminating; details at (:CSSNM00006:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log
    2011-07-02 18:17:01.122
    [cssd(28562)]CRS-1603:CSSD on node rac2 shutdown by user.
    2011-07-02 18:17:06.917
    [ohasd(27261)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac2'.
    2011-07-02 18:17:23.602
    [mdnsd(28485)]CRS-5602:mDNS service stopping by request.
    2011-07-02 18:17:36.217
    [gpnpd(28732)]CRS-2328:GPNPD started on node rac2.
    2011-07-02 18:17:43.673
    [cssd(28794)]CRS-1713:CSSD daemon is started in clustered mode
    2011-07-02 18:17:49.826
    [cssd(28794)]CRS-1707:Lease acquisition for node rac2 number 2 completed
    2011-07-02 18:17:49.865
    [cssd(28794)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
    2011-07-02 18:18:03.049
    [cssd(28794)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
    2011-07-02 18:18:06.160
    [ctssd(28861)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
    2011-07-02 18:18:06.220
    [ctssd(28861)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
    2011-07-02 18:18:06.238
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 18:18:06.239
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 18:18:06.794
    [ctssd(28861)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
    [client(28891)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(28895)]CRS-10001:ACFS-9322: done.
    2011-07-02 18:18:33.465
    [crsd(29020)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:33.575
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:35.757
    [crsd(29051)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:36.129
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:38.596
    [crsd(29066)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:39.146
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:41.058
    [crsd(29085)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:41.435
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:44.255
    [crsd(29101)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:45.165
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:47.013
    [crsd(29121)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:47.409
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:50.071
    [crsd(29136)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:50.118
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:51.843
    [crsd(29156)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:52.373
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:54.361
    [crsd(29171)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:54.772
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:56.620
    [crsd(29202)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:57.104
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:58.997
    [crsd(29218)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:59.301
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:59.302
    [ohasd(27261)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    2011-07-02 18:49:58.070
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 18:49:58.070
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 19:21:33.362
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 19:21:33.362
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 19:52:05.271
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 19:52:05.271
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 20:22:53.696
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 20:22:53.696
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 20:53:43.949
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 20:53:43.949
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 21:24:32.990
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 21:24:32.990
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 21:55:21.907
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 21:55:21.908
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 22:26:45.752
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 22:26:45.752
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 22:57:54.682
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 22:57:54.683
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 23:07:28.603
    [cssd(28794)]CRS-1612:Network communication with node rac1 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.020 seconds
    2011-07-02 23:07:35.621
    [cssd(28794)]CRS-1611:Network communication with node rac1 (1) missing for 75% of timeout interval.  Removal of this node from cluster in 7.010 seconds
    2011-07-02 23:07:39.629
    [cssd(28794)]CRS-1610:Network communication with node rac1 (1) missing for 90% of timeout interval.  Removal of this node from cluster in 3.000 seconds
    2011-07-02 23:07:42.641
    [cssd(28794)]CRS-1632:Node rac1 is being removed from the cluster in cluster incarnation 205080558
    2011-07-02 23:07:44.751
    [cssd(28794)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
    2011-07-02 23:07:45.326
    [ctssd(28861)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
    2011-07-04 19:46:26.008
    [ohasd(27261)]CRS-8011:reboot advisory message from host: rac1, component: mo155738, with time stamp: L-2011-07-04-19:44:43.318
    [ohasd(27261)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
    *[root@rac2 rac2]#* This log file start with complaint that OLR is not accessible. Here is what I see (rca2):
    -rw------- 1 root oinstall 272756736 Jul  2 18:18 /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olrAnd I guess rest of the problems start with this.

  • Grid installation: root.sh failed on the first node on Solaris cluster 4.1

    Hi all,
    I'm trying to install the Grid (11.2.0.3.0) on the 2 node-clusters (OSC 4.1).
    When I run the root.sh on the first node, I got the out put as follow:
    xha239080-root-5.11# root.sh
    Performing root user operation for Oracle 11g
    The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /Grid/CRShome
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
    /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]:
    Warning: /usr/local/bin is read only. No files will be copied.
    Creating /var/opt/oracle/oratab file...
    Entries will be added to the /var/opt/oracle/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root script.
    Now product-specific root actions will be performed.
    Using configuration parameter file: /Grid/CRShome/crs/install/crsconfig_params
    Creating trace directory
    User ignored Prerequisites during installation
    OLR initialization - successful
    root wallet
    root wallet cert
    root cert export
    peer wallet
    profile reader wallet
    pa wallet
    peer wallet keys
    pa wallet keys
    peer cert request
    pa cert request
    peer cert
    pa cert
    peer root cert TP
    profile reader root cert TP
    pa root cert TP
    peer pa cert TP
    pa peer cert TP
    profile reader pa cert TP
    profile reader peer cert TP
    peer user cert
    pa user cert
    Adding Clusterware entries to inittab
    CRS-2672: Attempting to start 'ora.mdnsd' on 'xha239080'
    CRS-2676: Start of 'ora.mdnsd' on 'xha239080' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'xha239080'
    CRS-2676: Start of 'ora.gpnpd' on 'xha239080' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'xha239080'
    CRS-2672: Attempting to start 'ora.gipcd' on 'xha239080'
    CRS-2676: Start of 'ora.cssdmonitor' on 'xha239080' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'xha239080' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'xha239080'
    CRS-2672: Attempting to start 'ora.diskmon' on 'xha239080'
    CRS-2676: Start of 'ora.diskmon' on 'xha239080' succeeded
    CRS-2676: Start of 'ora.cssd' on 'xha239080' succeeded
    ASM created and started successfully.
    Disk Group DATA created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    CRS-4256: Updating the profile
    Successful addition of voting disk 9cdb938773bc4f16bf332edac499fd06.
    Successful addition of voting disk 842907db11f74f59bf65247138d6e8f5.
    Successful addition of voting disk 748852d2a5c84f72bfcd50d60f65654d.
    Successfully replaced voting disk group with +DATA.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ## STATE File Universal Id File Name Disk group
    1. ONLINE 9cdb938773bc4f16bf332edac499fd06 (/dev/did/rdsk/d10s6) [DATA]
    2. ONLINE 842907db11f74f59bf65247138d6e8f5 (/dev/did/rdsk/d8s6) [DATA]
    3. ONLINE 748852d2a5c84f72bfcd50d60f65654d (/dev/did/rdsk/d9s6) [DATA]
    Located 3 voting disk(s).
    Start of resource "ora.cssd" failed
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'xha239080'
    CRS-2672: Attempting to start 'ora.gipcd' on 'xha239080'
    CRS-2676: Start of 'ora.cssdmonitor' on 'xha239080' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'xha239080' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'xha239080'
    CRS-2672: Attempting to start 'ora.diskmon' on 'xha239080'
    CRS-2676: Start of 'ora.diskmon' on 'xha239080' succeeded
    CRS-2674: Start of 'ora.cssd' on 'xha239080' failed
    CRS-2679: Attempting to clean 'ora.cssd' on 'xha239080'
    CRS-2681: Clean of 'ora.cssd' on 'xha239080' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'xha239080'
    CRS-2677: Stop of 'ora.gipcd' on 'xha239080' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'xha239080'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'xha239080' succeeded
    CRS-5804: Communication error with agent process
    CRS-4000: Command Start failed, or completed with errors.
    Failed to start Oracle Grid Infrastructure stack
    Failed to start Cluster Synchorinisation Service in clustered mode at /Grid/CRShome/crs/install/crsconfig_lib.pm line 1211.
    /Grid/CRShome/perl/bin/perl -I/Grid/CRShome/perl/lib -I/Grid/CRShome/crs/install /Grid/CRShome/crs/install/rootcrs.pl execution failed
    xha239080-root-5.11# history
    checking the ocssd.log, I see some thing as follow:
    2013-09-16 18:46:24.238: [    CSSD][1]clssscmain: Starting CSS daemon, version 11.2.0.3.0, in (clustered) mode with uniqueness value 1379371584
    2013-09-16 18:46:24.239: [    CSSD][1]clssscmain: Environment is production
    2013-09-16 18:46:24.239: [    CSSD][1]clssscmain: Core file size limit extended
    2013-09-16 18:46:24.248: [    CSSD][1]clssscmain: GIPCHA down 1
    2013-09-16 18:46:24.249: [    CSSD][1]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
    2013-09-16 18:46:24.250: [    CSSD][1]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
    2013-09-16 18:46:24.250: [    CSSD][1]clssscExtendLimits: The current soft limit for locked memory is 4294967293, hard limit is 4294967293
    2013-09-16 18:46:24.250: [    CSSD][1]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
    2013-09-16 18:46:24.250: [    CSSD][1]clssscSetPrivEnv: Setting priority to 4
    2013-09-16 18:46:24.253: [    CSSD][1]clssscSetPrivEnv: unable to set priority to 4
    2013-09-16 18:46:24.253: [    CSSD][1]SLOS: cat=-2, opn=scls_mem_lockdown, dep=11, loc=mlockall
    unable to lock memory
    2013-09-16 18:46:24.253: [    CSSD][1](:CSSSC00011:)clssscExit: A fatal error occurred during initialization
    Do anyone have any idea what going on and how can I fix it ?

    Hi,
    solaris has several issues with DISM, e.g.:
    Solaris 10 and Solaris 11 Shared Memory Locking May Fail (Doc ID 1590151.1)
    Sounds like Solaris Cluster  has a similar bug. A "workaround" is to reboot the (cluster) zone, that "fixes" the mlock error. This bug was introduced with updates in september, atleast to our environment (Solaris 11.1). Prior i did not have the issue and now i have to restart the entire zone, whenever i stop crs.
    With 11.2.0.3 the root.sh script can be rerun without prior cleaning up, so you should be able to continue installation at that point after the reboot. After the root.sh completes some configuration assistants need to be run, to complete the installation. You need to execute this manually as you wipe your oui session
    Kind Regards
    Thomas

  • Root.sh failed at second node OUL 6.3 Oracle GRID 11.2.0.3

    Hi, im installing a two node cluster mounted on Oracle Linux 6.3 with Oracle DB 11.2.0.3, the installation went smooth up until the execution of the root.sh script on the second node.
    THe script return this final lines:
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node nodo1, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    Start of resource "ora.crsd" failed
    CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'nodo2'
    CRS-4000: Command Start failed, or completed with errors.
    Failed to start Oracle Grid Infrastructure stack
    Failed to start Cluster Ready Services at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1286.
    /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
    In $GRID_HOME/log/node2/alertnode.log It appears to be a Cluster Time Synchronization Service issue, (i didn't synchronyze the nodes..) however the CTSS is running in observer mode, wich i believe it shouldn't affect the installation process. After that i lost it...there's an entry CRS-5018 indicating that an unused HAIP route was removed... and then, out of the blue: CRS-5818:Aborted command 'start' for resource 'ora.asm'. Some clarification will be deeply apreciated.
    Here's the complete log:
    2013-04-01 13:39:35.358
    [client(12163)]CRS-2101:The OLR was formatted using version 3.
    2013-04-01 19:40:19.597
    [ohasd(12338)]CRS-2112:The OLR service started on node nodo2.
    2013-04-01 19:40:19.657
    [ohasd(12338)]CRS-1301:Oracle High Availability Service started on node nodo2.
    [client(12526)]CRS-10001:01-Apr-13 13:41 ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.39-400.17.2.el6uek.i686'
    [client(12528)]CRS-10001:01-Apr-13 13:41 ACFS-9201: Not Supported
    [client(12603)]CRS-10001:01-Apr-13 13:41 ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.39-400.17.2.el6uek.i686'
    2013-04-01 19:41:17.509
    [ohasd(12338)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2013-04-01 19:41:17.618
    [gpnpd(12695)]CRS-2328:GPNPD started on node nodo2.
    2013-04-01 19:41:21.363
    [cssd(12755)]CRS-1713:CSSD daemon is started in exclusive mode
    2013-04-01 19:41:23.194
    [ohasd(12338)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
    2013-04-01 19:41:56.144
    [cssd(12755)]CRS-1707:Lease acquisition for node nodo2 number 2 completed
    2013-04-01 19:41:57.545
    [cssd(12755)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/ASM_DISK_1; details in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log.
    [cssd(12755)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node nodo1 and is terminating; details at (:CSSNM00006:) in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log
    2013-04-01 19:41:58.549
    [ohasd(12338)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'nodo2'.
    2013-04-01 19:42:10.025
    [gpnpd(12695)]CRS-2329:GPNPD on node nodo2 shutdown.
    2013-04-01 19:42:11.407
    [mdnsd(12685)]CRS-5602:mDNS service stopping by request.
    2013-04-01 19:42:29.642
    [gpnpd(12947)]CRS-2328:GPNPD started on node nodo2.
    2013-04-01 19:42:33.241
    [cssd(13012)]CRS-1713:CSSD daemon is started in clustered mode
    2013-04-01 19:42:35.104
    [ohasd(12338)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
    2013-04-01 19:42:44.065
    [cssd(13012)]CRS-1707:Lease acquisition for node nodo2 number 2 completed
    2013-04-01 19:42:45.484
    [cssd(13012)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/ASM_DISK_1; details in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log.
    2013-04-01 19:42:52.138
    [cssd(13012)]CRS-1601:CSSD Reconfiguration complete. Active nodes are nodo1 nodo2 .
    2013-04-01 19:42:55.081
    [ctssd(13076)]CRS-2403:The Cluster Time Synchronization Service on host nodo2 is in observer mode.
    2013-04-01 19:42:55.581
    [ctssd(13076)]CRS-2401:The Cluster Time Synchronization Service started on host nodo2.
    2013-04-01 19:42:55.581
    [ctssd(13076)]CRS-2407:The new Cluster Time Synchronization Service reference node is host nodo1.
    2013-04-01 19:43:08.875
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 19:43:08.876
    [ctssd(13076)]CRS-2409:The clock on host nodo2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2013-04-01 19:43:13.565
    [u01/app/11.2.0/grid/bin/orarootagent.bin(13064)]CRS-5018:(:CLSN00037:) Removed unused HAIP route: 169.254.0.0 / 255.255.0.0 / 0.0.0.0 / eth0
    2013-04-01 19:53:09.800
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5818:Aborted command 'start' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:0:223} in /u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log.
    2013-04-01 19:53:11.827
    [ohasd(12338)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) {0:0:223} in /u01/app/11.2.0/grid/log/nodo2/ohasd/ohasd.log.
    2013-04-01 19:53:12.779
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:53:13.892
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:53:43.877
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:54:13.891
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:54:43.906
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:55:13.914
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:55:43.918
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:56:13.922
    [u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    2013-04-01 19:56:53.209
    [crsd(13741)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:07:01.128
    [crsd(13741)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:07:01.278
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:07:08.689
    [crsd(15248)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:13:10.138
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 20:17:13.024
    [crsd(15248)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:17:13.171
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:17:20.826
    [crsd(16746)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:27:25.020
    [crsd(16746)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:27:25.176
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:27:31.591
    [crsd(18266)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:37:35.668
    [crsd(18266)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:37:35.808
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:37:43.209
    [crsd(19762)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:43:11.160
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 20:47:47.487
    [crsd(19762)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:47:47.637
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:47:55.086
    [crsd(21242)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 20:57:59.343
    [crsd(21242)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 20:57:59.492
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 20:58:06.996
    [crsd(22744)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:08:11.046
    [crsd(22744)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:08:11.192
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:08:18.726
    [crsd(24260)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:13:12.000
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 21:18:22.262
    [crsd(24260)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:18:22.411
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:18:29.927
    [crsd(25759)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:28:34.467
    [crsd(25759)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:28:34.616
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:28:41.990
    [crsd(27291)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:38:45.012
    [crsd(27291)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:38:45.160
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:38:52.790
    [crsd(28784)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:43:12.378
    [ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
    2013-04-01 21:48:56.285
    [crsd(28784)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:48:56.435
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:49:04.421
    [crsd(30272)]CRS-1012:The OCR service started on node nodo2.
    2013-04-01 21:59:08.183
    [crsd(30272)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
    2013-04-01 21:59:08.318
    [ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
    2013-04-01 21:59:15.860
    [crsd(31772)]CRS-1012:The OCR service started on node nodo2.

    Hi santysharma, thanks for the reply, i have two ethernet interfaces: eth0 (public network 192.168.1.0) and eth1 (private network 10.5.3.0), there is no device using that ip range, here's the output of route command:
    (Sorry for the alignment, i tried to tab it but the editor trims it again)
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    default 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
    private * 255.255.255.0 U 0 0 0 eth1
    link-local * 255.255.0.0 U 1002 0 0 eth0
    link-local * 255.255.0.0 U 1003 0 0 eth1
    public * 255.255.255.0 U 0 0 0 eth0
    And the /etc/hosts file
    127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
    10.5.3.1 nodo1.cluster nodo1
    10.5.3.2 nodo2.cluster nodo2
    192.168.1.13 cluster-scan
    192.168.1.14 nodo1-vip
    192.168.1.15 nodo2-vip
    And the ifconfig -a
    eth0 Link encap:Ethernet HWaddr C8:3A:35:D9:C6:2B
    inet addr:192.168.1.12 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::ca3a:35ff:fed9:c62b/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:34708 errors:0 dropped:18 overruns:0 frame:0
    TX packets:24693 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:48545969 (46.2 MiB) TX bytes:1994381 (1.9 MiB)
    eth1 Link encap:Ethernet HWaddr 00:0D:87:D0:A3:8E
    inet addr:10.5.3.2 Bcast:10.5.3.255 Mask:255.255.255.0
    inet6 addr: fe80::20d:87ff:fed0:a38e/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:0 (0.0 b) TX bytes:5344 (5.2 KiB)
    Interrupt:23 Base address:0x6000
    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:20 errors:0 dropped:0 overruns:0 frame:0
    TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:1320 (1.2 KiB) TX bytes:1320 (1.2 KiB)
    Now that i'm thinking i've read somewhere that ipv6 was no supported...yet there's no relation with the 169.254.x.x ip range.

  • Root.sh fails on second node

    I already posted this issue on database installation forum, and was suggested to post it on this forum.
    Here are the details.
    I am running Linux 64bit on ESx clients. Installing Oracle 11gR2.
    It passed all the per-requisite. Run root.sh on first node. It finished with no errorrs.
    On second node I got the following:
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /u01/app/11.2.0/grid
    Enter the full pathname of the local bin directory: [usr/local/bin]:
    The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
    [n]:
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2010-07-13 12:51:28: Parsing the host name
    2010-07-13 12:51:28: Checking for super user privileges
    2010-07-13 12:51:28: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node fred0224, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'fred0225'
    CRS-2676: Start of 'ora.mdnsd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'fred0225'
    CRS-2676: Start of 'ora.gipcd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'fred0225'
    CRS-2676: Start of 'ora.gpnpd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'fred0225'
    CRS-2676: Start of 'ora.cssdmonitor' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'fred0225'
    CRS-2672: Attempting to start 'ora.diskmon' on 'fred0225'
    CRS-2676: Start of 'ora.diskmon' on 'fred0225' succeeded
    CRS-2676: Start of 'ora.cssd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'fred0225'
    Start action for octssd aborted
    CRS-2676: Start of 'ora.ctssd' on 'fred0225' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'fred0225'
    CRS-2672: Attempting to start 'ora.asm' on 'fred0225'
    CRS-2676: Start of 'ora.drivers.acfs' on 'fred0225' succeeded
    CRS-2676: Start of 'ora.asm' on 'fred0225' succeeded
    CRS-2664: Resource 'ora.ctssd' is already running on 'fred0225'
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.asm -init
    Start of resource "ora.asm -init" failed
    Failed to start ASM
    Failed to start Oracle Clusterware stack
    In the ocssd.log I found
    [ CSSD][3559689984]clssnmvDHBValidateNCopy: node 1, fred0224, has a disk HB, but no network HB, DHB has rcfg 174483948, wrtcnt, 232, LATS 521702664, lastSeqNo 232, uniqueness 1279039649, timestamp 1279039959/521874274
    In oraagent_oracle.log I found
    [ clsdmc][1212365120]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD)) with status 9
    2010-07-13 12:54:07.234: [ora.gpnpd][1212365120] [check] Error = error 9 encountered when connecting to GPNPD
    2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Calling PID check for daemon
    2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Trying to check PID = 20584
    2010-07-13 12:54:07.432: [ COMMCRS][1285794112]clsc_connect: (0x1304d850) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD))
    [ clsdmc][1222854976]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD)) with status 9
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Error = error 9 encountered when connecting to MDNSD
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Calling PID check for daemon
    2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Trying to check PID = 20571
    2010-07-13 12:54:08.841: [ COMMCRS][1201875264]clsc_connect: (0x12f3b1d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD))
    [ clsdmc][1159915840]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD)) with status 9
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Error = error 9 encountered when connecting to GIPCD
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Calling PID check for daemon
    2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Trying to check PID = 20566
    2010-07-13 12:54:10.242: [ COMMCRS][1254324544]clsc_connect: (0x12f35630) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD))
    In oracssdagent_root.log I found
    2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clssscConnect: gipc request failed with 29 (0x16)
    2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clsssInitNative: connect failed, rc 29
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clssnsqlnum: RPC failed rc 3
    2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_cssini: failed 3 to fetch node number
    2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_init: css init done, nodenum -1.
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssRecvMsg: got a disconnect from the server while waiting for message type 43
    2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssGetNLSData: Failure receiving a msg, rc 3
    If you need more info, let me know.

    Well, the error clearly indicates that a communication problem exists on the private interconnect.
    Could this be a setting in ESX, which prevents some communication between the clients on the second network card? Any routing table in ESX not configured correctly?
    Sebastian

  • Root.ah fails on 2nd node(rac2) with [ ORA-15018,ORA-15017,ORA-15003 ]

    Hi All,
    I m trying to setup 11gR2 Grid installation on two-node Rac . When it comes to running root.sh on second node (i.e. rac2) it fails with below error. Could please anyone help me out. This is my 3rd attempt and all fails with below errors on node 2.
    rac2:
    [root@rac2 grid_home]# ./root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= grid
        ORACLE_HOME=  /u01/grid_home
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
       Copying dbhome to /usr/local/bin ...
       Copying oraenv to /usr/local/bin ...
       Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2013-07-10 18:53:15: Parsing the host name
    2013-07-10 18:53:15: Checking for super user privileges
    2013-07-10 18:53:15: User has super user privileges
    Using configuration parameter file: /u01/grid_home/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
    CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
    DiskGroup CRS creation failed with the following message:
    ORA-15018: diskgroup cannot be created
    ORA-15017: diskgroup "CRS" cannot be mounted
    ORA-15003: diskgroup "CRS" already mounted in another lock name space
    Configuration of ASM failed, see logs for details
    Did not succssfully configure and start ASM
    CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/grid_home/bin/crsctl stop resource ora.crsd -init
    Stop of resource "ora.crsd -init" failed
    Failed to stop CRSD
    CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
    CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
    CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac2'
    CRS-2677: Stop of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
    CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
    CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
    CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
    CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
    Initial cluster configuration failed.  See /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log for details
    [root@rac2 grid_home]#
    rac2  alertrac2.log
    [root@rac2 rac2]# cat -n alertrac2.log
         1  Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
         2  2013-07-10 18:53:16.145
         3  [client(13088)]CRS-2106:The OLR location /u01/grid_home/cdata/rac2.olr is inaccessible. Details in /u01/grid_home/log/rac2/client/ocrconfig_13088.log.
         4  2013-07-10 18:53:16.228
         5  [client(13088)]CRS-2101:The OLR was formatted using version 3.
         6  2013-07-10 18:53:31.734
         7  [ohasd(13132)]CRS-2112:The OLR service started on node rac2.
         8  2013-07-10 18:53:31.893
         9  [ohasd(13132)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
        10  2013-07-10 18:53:53.762
        11  [ohasd(13132)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
        12  2013-07-10 18:53:55.381
        13  [cssd(14409)]CRS-1713:CSSD daemon is started in exclusive mode
        14  2013-07-10 18:54:01.530
        15  [cssd(14409)]CRS-1709:Lease acquisition failed for node rac2 because no voting file has been configured; Details at (:CSSNM00031:) in /u01/grid_home/log/rac2/cssd/ocssd.log
        16  2013-07-10 18:54:19.113
        17  [cssd(14409)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
        18  2013-07-10 18:54:19.910
        19  [ctssd(14465)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
        20  2013-07-10 18:54:19.920
        21  [ctssd(14465)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
        22  2013-07-10 18:54:20.903
        23  [ctssd(14465)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
        24  [client(14715)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
        25  [client(14719)]CRS-10001:ACFS-9322: done.
        26  2013-07-10 18:54:47.104
        27  [ctssd(14465)]CRS-2405:The Cluster Time Synchronization Service on host rac2 is shutdown by user
        28  2013-07-10 18:54:55.837
        29  [cssd(14409)]CRS-1603:CSSD on node rac2 shutdown by user.
    rac2 rootcrs logfile
    [root@rac2 rac2]# cat  /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log
    2013-07-10 18:53:15: The configuration parameter file /u01/grid_home/crs/install/crsconfig_params is valid
    2013-07-10 18:53:15: Checking for super user privileges
    2013-07-10 18:53:15: User has super user privileges
    2013-07-10 18:53:15: ### Printing the configuration values from files:
    2013-07-10 18:53:15:    /u01/grid_home/crs/install/crsconfig_params
    2013-07-10 18:53:15:    /u01/grid_home/crs/install/s_crsconfig_defs
    2013-07-10 18:53:15: ASM_DISCOVERY_STRING=
    2013-07-10 18:53:15: ASM_DISKS=ORCL:CRS1
    2013-07-10 18:53:15: ASM_DISK_GROUP=CRS
    2013-07-10 18:53:15: ASM_REDUNDANCY=EXTERNAL
    2013-07-10 18:53:15: ASM_SPFILE=
    2013-07-10 18:53:15: ASM_UPGRADE=false
    2013-07-10 18:53:15: CLSCFG_MISSCOUNT=
    2013-07-10 18:53:15: CLUSTER_GUID=
    2013-07-10 18:53:15: CLUSTER_NAME=rac-scan
    2013-07-10 18:53:15: CRS_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'
    2013-07-10 18:53:15: CRS_STORAGE_OPTION=1
    2013-07-10 18:53:15: CSS_LEASEDURATION=400
    2013-07-10 18:53:15: DIRPREFIX=
    2013-07-10 18:53:15: DISABLE_OPROCD=0
    2013-07-10 18:53:15: EMBASEJAR_NAME=oemlt.jar
    2013-07-10 18:53:15: EWTJAR_NAME=ewt3.jar
    2013-07-10 18:53:15: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
    2013-07-10 18:53:15: GNS_ADDR_LIST=
    2013-07-10 18:53:15: GNS_ALLOW_NET_LIST=
    2013-07-10 18:53:15: GNS_CONF=false
    2013-07-10 18:53:15: GNS_DENY_ITF_LIST=
    2013-07-10 18:53:15: GNS_DENY_NET_LIST=
    2013-07-10 18:53:15: GNS_DOMAIN_LIST=
    2013-07-10 18:53:15: GPNPCONFIGDIR=/u01/grid_home
    2013-07-10 18:53:15: GPNPGCONFIGDIR=/u01/grid_home
    2013-07-10 18:53:15: GPNP_PA=
    2013-07-10 18:53:15: HELPJAR_NAME=help4.jar
    2013-07-10 18:53:15: HOST_NAME_LIST=rac1,rac2
    2013-07-10 18:53:15: ID=/etc/init.d
    2013-07-10 18:53:15: INIT=/sbin/init
    2013-07-10 18:53:15: IT=/etc/inittab
    2013-07-10 18:53:15: JEWTJAR_NAME=jewt4.jar
    2013-07-10 18:53:15: JLIBDIR=/u01/grid_home/jlib
    2013-07-10 18:53:15: JREDIR=/u01/grid_home/jdk/jre/
    2013-07-10 18:53:15: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
    2013-07-10 18:53:15: MSGFILE=/var/adm/messages
    2013-07-10 18:53:15: NETCFGJAR_NAME=netcfg.jar
    2013-07-10 18:53:15: NETWORKS="eth0"/192.168.0.0:public,"eth1"/192.168.1.0:cluster_interconnect
    2013-07-10 18:53:15: NEW_HOST_NAME_LIST=
    2013-07-10 18:53:15: NEW_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'
    2013-07-10 18:53:15: NEW_NODE_NAME_LIST=
    2013-07-10 18:53:15: NEW_PRIVATE_NAME_LIST=
    2013-07-10 18:53:15: NODELIST=rac1,rac2
    2013-07-10 18:53:15: NODE_NAME_LIST=rac1,rac2
    2013-07-10 18:53:15: OCFS_CONFIG=
    2013-07-10 18:53:15: OCRCONFIG=/etc/oracle/ocr.loc
    2013-07-10 18:53:15: OCRCONFIGDIR=/etc/oracle
    2013-07-10 18:53:15: OCRID=
    2013-07-10 18:53:15: OCRLOC=ocr.loc
    2013-07-10 18:53:15: OCR_LOCATIONS=NO_VAL
    2013-07-10 18:53:15: OLASTGASPDIR=/etc/oracle/lastgasp
    2013-07-10 18:53:15: OLRCONFIG=/etc/oracle/olr.loc
    2013-07-10 18:53:15: OLRCONFIGDIR=/etc/oracle
    2013-07-10 18:53:15: OLRLOC=olr.loc
    2013-07-10 18:53:15: OPROCDCHECKDIR=/etc/oracle/oprocd/check
    2013-07-10 18:53:15: OPROCDDIR=/etc/oracle/oprocd
    2013-07-10 18:53:15: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
    2013-07-10 18:53:15: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
    2013-07-10 18:53:15: ORACLE_BASE=/u01/11.2.0
    2013-07-10 18:53:15: ORACLE_HOME=/u01/grid_home
    2013-07-10 18:53:15: ORACLE_OWNER=grid
    2013-07-10 18:53:15: ORA_ASM_GROUP=asmadmin
    2013-07-10 18:53:15: ORA_DBA_GROUP=oinstall
    2013-07-10 18:53:15: PRIVATE_NAME_LIST=
    2013-07-10 18:53:15: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
    2013-07-10 18:53:15: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
    2013-07-10 18:53:15: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
    2013-07-10 18:53:15: RC_KILL=K19
    2013-07-10 18:53:15: RC_KILL_OLD=K96
    2013-07-10 18:53:15: RC_START=S96
    2013-07-10 18:53:15: SCAN_NAME=rac-scan.naveed.com
    2013-07-10 18:53:15: SCAN_PORT=1521
    2013-07-10 18:53:15: SCRBASE=/etc/oracle/scls_scr
    2013-07-10 18:53:15: SHAREJAR_NAME=share.jar
    2013-07-10 18:53:15: SILENT=false
    2013-07-10 18:53:15: SO_EXT=so
    2013-07-10 18:53:15: SRVCFGLOC=srvConfig.loc
    2013-07-10 18:53:15: SRVCONFIG=/var/opt/oracle/srvConfig.loc
    2013-07-10 18:53:15: SRVCONFIGDIR=/var/opt/oracle
    2013-07-10 18:53:15: VNDR_CLUSTER=false
    2013-07-10 18:53:15: VOTING_DISKS=NO_VAL
    2013-07-10 18:53:15: ### Printing other configuration values ###
    2013-07-10 18:53:15: CLSCFG_EXTRA_PARMS=
    2013-07-10 18:53:15: CRSDelete=0
    2013-07-10 18:53:15: CRSPatch=0
    2013-07-10 18:53:15: DEBUG=
    2013-07-10 18:53:15: DOWNGRADE=
    2013-07-10 18:53:15: HAS_GROUP=oinstall
    2013-07-10 18:53:15: HAS_USER=root
    2013-07-10 18:53:15: HOST=rac2
    2013-07-10 18:53:15: IS_SIHA=0
    2013-07-10 18:53:15: OLR_DIRECTORY=/u01/grid_home/cdata
    2013-07-10 18:53:15: OLR_LOCATION=/u01/grid_home/cdata/rac2.olr
    2013-07-10 18:53:15: ORA_CRS_HOME=/u01/grid_home
    2013-07-10 18:53:15: SUPERUSER=root
    2013-07-10 18:53:15: UPGRADE=
    2013-07-10 18:53:15: VF_DISCOVERY_STRING=
    2013-07-10 18:53:15: addfile=/u01/grid_home/crs/install/crsconfig_addparams
    2013-07-10 18:53:15: crscfg_trace=1
    2013-07-10 18:53:15: crscfg_trace_file=/u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log
    2013-07-10 18:53:15: hosts=
    2013-07-10 18:53:15: oldcrshome=
    2013-07-10 18:53:15: oldcrsver=
    2013-07-10 18:53:15: osdfile=/u01/grid_home/crs/install/s_crsconfig_defs
    2013-07-10 18:53:15: parameters_valid=1
    2013-07-10 18:53:15: paramfile=/u01/grid_home/crs/install/crsconfig_params
    2013-07-10 18:53:15: platform_family=unix
    2013-07-10 18:53:15: srvctl_trc_suff=0
    2013-07-10 18:53:15: unlock_crshome=
    2013-07-10 18:53:15: user_is_superuser=1
    2013-07-10 18:53:15: ### Printing of configuration values complete ###
    2013-07-10 18:53:15: Oracle CRS stack is not configured yet
    2013-07-10 18:53:15: CRS is not yet configured. Hence, will proceed to configure CRS
    2013-07-10 18:53:15: Cluster-wide one-time actions... Done!
    2013-07-10 18:53:15: Oracle CRS home = /u01/grid_home
    2013-07-10 18:53:15: Host name = rac2
    2013-07-10 18:53:15: CRS user = grid
    2013-07-10 18:53:15: Oracle CRS home = /u01/grid_home
    2013-07-10 18:53:15: GPnP host = rac2
    2013-07-10 18:53:15: Oracle GPnP home = /u01/grid_home/gpnp
    2013-07-10 18:53:15: Oracle GPnP local home = /u01/grid_home/gpnp/rac2
    2013-07-10 18:53:15: GPnP directories verified.
    2013-07-10 18:53:15: Checking to see if Oracle CRS stack is already configured
    2013-07-10 18:53:15: Oracle CRS stack is not configured yet
    2013-07-10 18:53:15: ---Checking local gpnp setup...
    2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml" does not exist
    2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso" does not exist
    2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso" does not exist
    2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp/rac2: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
    2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp/rac2: INVALID (bad profile/wallet)
    2013-07-10 18:53:15: ---Checking cluster-wide gpnp setup...
    2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1
    2013-07-10 18:53:15: gpnptool: run /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer
    2013-07-10 18:53:15: Running as user grid: /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer
    2013-07-10 18:53:15: s_run_as_user2: Running /bin/su grid -c ' /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer '
    2013-07-10 18:53:15: Removing file /tmp/file0qKE0c
    2013-07-10 18:53:15: Successfully removed file: /tmp/file0qKE0c
    2013-07-10 18:53:15: /bin/su successfully executed
    2013-07-10 18:53:15: gpnptool: rc=0
    2013-07-10 18:53:15: gpnptool output:
    Profile signature is valid.
    2013-07-10 18:53:15: Profile "/u01/grid_home/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/grid_home/gpnp/wallets/peer"
    2013-07-10 18:53:15: gpnptool: run /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer
    2013-07-10 18:53:15: Running as user grid: /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer
    2013-07-10 18:53:15: s_run_as_user2: Running /bin/su grid -c ' /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer '
    2013-07-10 18:53:16: Removing file /tmp/filebkOtBv
    2013-07-10 18:53:16: Successfully removed file: /tmp/filebkOtBv
    2013-07-10 18:53:16: /bin/su successfully executed
    2013-07-10 18:53:16: gpnptool: rc=0
    2013-07-10 18:53:16: gpnptool output:
    Profile signature is valid.
    2013-07-10 18:53:16: Profile "/u01/grid_home/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/grid_home/gpnp/wallets/prdr"
    2013-07-10 18:53:16: chk gpnphome /u01/grid_home/gpnp: OK
    2013-07-10 18:53:16: GPnP Wallets ownership/permissions successfully set.
    2013-07-10 18:53:16: gpnp setup checked: local valid? 0 cluster-wide valid? 1
    2013-07-10 18:53:16: Taking cluster-wide setup as local
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/profiles/peer/profile.xml" => "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/peer/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/prdr/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/profiles/peer/profile_orig.xml" => "/u01/grid_home/gpnp/rac2/profiles/peer/profile_orig.xml"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/profiles/peer/profile_orig.xml" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/root/ewallet.p12" => "/u01/grid_home/gpnp/rac2/wallets/root/ewallet.p12"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/root/ewallet.p12" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/pa/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/pa/cwallet.sso"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/pa/cwallet.sso" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/root/b64certificate.txt" => "/u01/grid_home/gpnp/rac2/wallets/root/b64certificate.txt"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/root/b64certificate.txt" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/peer/cert.txt" => "/u01/grid_home/gpnp/rac2/wallets/peer/cert.txt"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/peer/cert.txt" => (grid,oinstall)
    2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/pa/cert.txt" => "/u01/grid_home/gpnp/rac2/wallets/pa/cert.txt"
    2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/pa/cert.txt" => (grid,oinstall)
    2013-07-10 18:53:16: GPnP Wallets ownership/permissions successfully set.
    2013-07-10 18:53:16: gpnp setup: GOTCLUSTERWIDE
    2013-07-10 18:53:16: Validating for SI-CSS configuration
    2013-07-10 18:53:16: Retrieving OCR main disk location
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc
    2013-07-10 18:53:16: Unable to retrieve ocr disk info
    2013-07-10 18:53:16: Checking to see if any 9i GSD is up
    2013-07-10 18:53:16: libskgxnBase_lib = /etc/ORCLcluster/oracm/lib/libskgxn2.so
    2013-07-10 18:53:16: libskgxn_lib = /opt/ORCLcluster/lib/libskgxn2.so
    2013-07-10 18:53:16: SKGXN library file does not exists
    2013-07-10 18:53:16: OLR location = /u01/grid_home/cdata/rac2.olr
    2013-07-10 18:53:16: Oracle CRS Home = /u01/grid_home
    2013-07-10 18:53:16: Validating /etc/oracle/olr.loc file for OLR location /u01/grid_home/cdata/rac2.olr
    2013-07-10 18:53:16: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig
    2013-07-10 18:53:16: Oracle CRS home = /u01/grid_home
    2013-07-10 18:53:16: Oracle cluster name = rac-scan
    2013-07-10 18:53:16: OCR locations = +CRS
    2013-07-10 18:53:16: Validating OCR
    2013-07-10 18:53:16: Retrieving OCR location used by previous installations
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrmirrorconfig_loc
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc3
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc4
    2013-07-10 18:53:16: Opening file OCRCONFIG
    2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc5
    2013-07-10 18:53:16: Checking if OCR sync file exists
    2013-07-10 18:53:16: No need to sync OCR file
    2013-07-10 18:53:16: OCR_LOCATION=+CRS
    2013-07-10 18:53:16: OCR_MIRROR_LOCATION=
    2013-07-10 18:53:16: OCR_MIRROR_LOC3=
    2013-07-10 18:53:16: OCR_MIRROR_LOC4=
    2013-07-10 18:53:16: OCR_MIRROR_LOC5=
    2013-07-10 18:53:16: Current OCR location=
    2013-07-10 18:53:16: Current OCR mirror location=
    2013-07-10 18:53:16: Current OCR mirror loc3=
    2013-07-10 18:53:16: Current OCR mirror loc4=
    2013-07-10 18:53:16: Current OCR mirror loc5=
    2013-07-10 18:53:16: Verifying current OCR settings with user entered values
    2013-07-10 18:53:16: Setting OCR locations in /etc/oracle/ocr.loc
    2013-07-10 18:53:16: Validating OCR locations in /etc/oracle/ocr.loc
    2013-07-10 18:53:16: Checking for existence of /etc/oracle/ocr.loc
    2013-07-10 18:53:16: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig
    2013-07-10 18:53:16: Setting ocr location +CRS
    2013-07-10 18:53:16: Creating or upgrading Oracle Local Registry (OLR)
    2013-07-10 18:53:16: OLR successfully created or upgraded
    2013-07-10 18:53:16: /u01/grid_home/bin/clscfg -localadd
    2013-07-10 18:53:16: Keys created in the OLR successfully
    2013-07-10 18:53:16: GPnP setup state: new-cluster-wide
    2013-07-10 18:53:16: GPnP cluster configuration already performed
    2013-07-10 18:53:16: Registering ohasd
    2013-07-10 18:53:16: init file = /u01/grid_home/crs/init/init.ohasd
    2013-07-10 18:53:16: Copying file /u01/grid_home/crs/init/init.ohasd to /etc/init.d directory
    2013-07-10 18:53:16: Setting init.ohasd permission in /etc/init.d directory
    2013-07-10 18:53:16: init file = /u01/grid_home/crs/init/ohasd
    2013-07-10 18:53:16: Copying file /u01/grid_home/crs/init/ohasd to /etc/init.d directory
    2013-07-10 18:53:16: Setting ohasd permission in /etc/init.d directory
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc3.d/S96ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc3.d/S96ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc5.d/S96ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc5.d/S96ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc0.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc0.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc1.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc1.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc2.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc2.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc4.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc4.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: Removing "/etc/rc.d/rc6.d/K19ohasd"
    2013-07-10 18:53:16: Removing file /etc/rc.d/rc6.d/K19ohasd
    2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
    2013-07-10 18:53:16: Failed to remove file:
    2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-10 18:53:16: The file ohasd has been successfully linked to the RC directories
    2013-07-10 18:53:16: Starting ohasd
    2013-07-10 18:53:16: itab entries=
    2013-07-10 18:53:21: Created backup /etc/inittab.no_crs
    2013-07-10 18:53:21: Appending to /etc/inittab.tmp:
    2013-07-10 18:53:21: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
    2013-07-10 18:53:21: Done updating /etc/inittab.tmp
    2013-07-10 18:53:21: Saved /etc/inittab.crs
    2013-07-10 18:53:21: Installed new /etc/inittab
    2013-07-10 18:53:36: ohasd is starting
    2013-07-10 18:53:36: Checking ohasd
    2013-07-10 18:53:37: ohasd started successfully
    2013-07-10 18:53:37: Creating CRS resources and dependencies
    2013-07-10 18:53:37: Configuring HASD
    2013-07-10 18:53:37: Registering type ora.daemon.type
    2013-07-10 18:53:37: Registering type ora.mdns.type
    2013-07-10 18:53:37: Registering type ora.gpnp.type
    2013-07-10 18:53:38: Registering type ora.gipc.type
    2013-07-10 18:53:38: Registering type ora.cssd.type
    2013-07-10 18:53:38: Registering type ora.cssdmonitor.type
    2013-07-10 18:53:39: Registering type ora.crs.type
    2013-07-10 18:53:39: Registering type ora.evm.type
    2013-07-10 18:53:39: Registering type ora.ctss.type
    2013-07-10 18:53:40: Registering type ora.asm.type
    2013-07-10 18:53:40: Registering type ora.drivers.acfs.type
    2013-07-10 18:53:40: Registering type ora.diskmon.type
    2013-07-10 18:53:51: ADVM/ACFS is configured
    2013-07-10 18:53:51: Successfully created CRS resources for cluster daemon and ASM
    2013-07-10 18:53:51: Checking if initial configuration has been performed
    2013-07-10 18:53:51: Starting CSS in exclusive mode
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    2013-07-10 18:54:19: CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    2013-07-10 18:54:19: Querying for existing CSS voting disks
    2013-07-10 18:54:19: Performing initial configuration for cluster
    2013-07-10 18:54:21: Start of resource "ora.ctssd -init" Succeeded
    2013-07-10 18:54:21: Configuring ASM via ASMCA
    2013-07-10 18:54:21: Executing as grid: /u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM
    2013-07-10 18:54:21: Running as user grid: /u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM
    2013-07-10 18:54:21:   Invoking "/u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM" as user "grid"
    2013-07-10 18:54:40: Configuration of ASM failed, see logs for details
    2013-07-10 18:54:40: Did not succssfully configure and start ASM
    2013-07-10 18:54:40: Exiting exclusive mode
    2013-07-10 18:54:40: Command return code of 1 (256) from command: /u01/grid_home/bin/crsctl stop resource ora.crsd -init
    2013-07-10 18:54:40: Stop of resource "ora.crsd -init" failed
    2013-07-10 18:54:40: Failed to stop CRSD
    2013-07-10 18:55:04: Initial cluster configuration failed.  See /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log for details
    Also below are some of the configs related to rac2 node
    [root@rac2 rac2]# rpm -qa | grep oracleasm
    oracleasmlib-2.0.4-1.el5
    oracleasm-support-2.1.8-1.el5
    oracleasm-2.6.18-274.el5xen-2.0.5-1.el5
    oracleasm-2.6.18-274.el5-2.0.5-1.el5
    oracleasm-2.6.18-274.el5debug-2.0.5-1.el5
    oracleasm-2.6.18-274.el5-debuginfo-2.0.5-1.el5
    [root@rac2 rac2]# /usr/sbin/oracleasm configure
    ORACLEASM_ENABLED=true
    ORACLEASM_UID=grid
    ORACLEASM_GID=asmadmin
    ORACLEASM_SCANBOOT=true
    ORACLEASM_SCANORDER=""
    ORACLEASM_SCANEXCLUDE=""
    ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"
    [root@rac2 rac2]# /usr/sbin/oracleasm status
    Checking if ASM is loaded: yes
    Checking if /dev/oracleasm is mounted: yes
    [root@rac2 rac2]# /usr/sbin/oracleasm listdisks
    CRS1
    DATA1
    FRA1
    [root@rac2 rac2]# ls -l /dev/oracleasm/disks/
    total 0
    brw-rw---- 1 grid asmadmin 8, 17 Jul 10 18:35 CRS1
    brw-rw---- 1 grid asmadmin 8, 33 Jul 10 18:36 DATA1
    brw-rw---- 1 grid asmadmin 8, 49 Jul 10 18:36 FRA1
    [root@rac2 rac2]# cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1               localhost.localdomain localhost
    ::1             localhost6.localdomain6 localhost6
    #Public IP's(eth0)
    192.168.0.101    rac1.naveed.com    rac1
    192.168.0.102    rac2.naveed.com    rac2
    #Private IP's(eth1)
    192.168.1.101    rac1-prv.naveed.com   rac1-prv
    192.168.1.102    rac2-prv.naveed.com   rac2-prv
    #VIPS
    192.168.0.221    rac1-vip.naveed.com   rac1-vip
    192.168.0.222    rac2-vip.naveed.com   rac2-vip
    #DNS server IP
    192.168.0.10    naveeddns.naveed.com   naveeddns
    [root@rac2 rac2]#
    Thanks in advance

    Hi,
    First of all thanks a lot for the response. You wont't beleive this is my 7th fresh installation and everytime in node 2 i m hit with this same error.
    Also i tried below procedure instead of fresh installation
    once i deconfig & rerun (./rootcrs.pl -verbose -deconfig -force) on node 2
    Using configuration parameter file: ./crsconfig_params
    PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
    PRCR-1068 : Failed to query resources
    Cannot communicate with crsd
    PRCR-1070 : Failed to check if resource ora.gsd is registered
    Cannot communicate with crsd
    PRCR-1070 : Failed to check if resource ora.ons is registered
    Cannot communicate with crsd
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4000: Command Stop failed, or completed with errors.
    CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
    CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac2'
    CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
    CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
    CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
    CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'
    CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
    CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
    CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
    CRS-2677: Stop of 'ora.drivers.acfs' on 'rac2' succeeded
    CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
    CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
    CRS-4133: Oracle High Availability Services has been stopped.
    Successfully deconfigured Oracle clusterware stack on this node
    [root@rac2 grid_home]# ./root.sh
    Performing root user operation for Oracle 11g
    The following environment variables are set as:
        ORACLE_OWNER= grid
        ORACLE_HOME=  /u01/grid_home
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
    The contents of "dbhome" have not changed. No need to overwrite.
    The contents of "oraenv" have not changed. No need to overwrite.
    The contents of "coraenv" have not changed. No need to overwrite.
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root script.
    Now product-specific root actions will be performed.
    Using configuration parameter file: /u01/grid_home/crs/install/crsconfig_params
    User ignored Prerequisites during installation
    OLR initialization - successful
    Adding Clusterware entries to inittab
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    ASM created and started successfully.
    Disk Group CRS mounted successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Successful addition of voting disk 636af26485ef4f27bfec31523aaa0660.
    Successfully replaced voting disk group with +CRS.
    CRS-4266: Voting file(s) successfully replaced
    ##  STATE    File Universal Id                File Name Disk group
    1. ONLINE   636af26485ef4f27bfec31523aaa0660 (ORCL:CRS1) [CRS]
    Located 1 voting disk(s).
    Start of resource "ora.crsd" failed
    CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'rac2'
    CRS-4000: Command Start failed, or completed with errors.
    Failed to start Oracle Grid Infrastructure stack
    Failed to start Cluster Ready Services at /u01/grid_home/crs/install/crsconfig_lib.pm line 1286.
    /u01/grid_home/perl/bin/perl -I/u01/grid_home/perl/lib -I/u01/grid_home/crs/install /u01/grid_home/crs/install/rootcrs.pl execution failed

  • Root.sh fails to bring up crs

    Hi All,
    I am installing 11gR2 on IBM AIX.
    Every ran fine. when i ran root.sh
    CRS-2676: Start of 'ora.diskmon' on 't24db1' succeeded
    CRS-2676: Start of 'ora.cssd' on 't24db1' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 't24db1'
    CRS-2676: Start of 'ora.ctssd' on 't24db1' succeeded
    ASM created and started successfully.
    DiskGroup OCR_VOTE created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'system'..
    Operation successful.
    CRS-2672: Attempting to start 'ora.crsd' on 't24db1'
    CRS-2676: Start of 'ora.crsd' on 't24db1' succeeded
    CRS-4256: Updating the profile
    Successful addition of voting disk 70e7b3c678074f2bbfcf933764645975.
    Successful addition of voting disk a85fe5cb0c9a4f88bf5adf830732d62f.
    Successful addition of voting disk 52d343aad6374f79bf0257908be9a6c9.
    Successfully replaced voting disk group with +OCR_VOTE.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ## STATE File Universal Id File Name Disk group
    1. ONLINE 70e7b3c678074f2bbfcf933764645975 (/dev/rhdisk10) [OCR_VOTE]
    2. ONLINE a85fe5cb0c9a4f88bf5adf830732d62f (/dev/rhdisk11) [OCR_VOTE]
    3. ONLINE 52d343aad6374f79bf0257908be9a6c9 (/dev/rhdisk5) [OCR_VOTE]
    Located 3 voting disk(s).
    CRS-2673: Attempting to stop 'ora.crsd' on 't24db1'
    CRS-2677: Stop of 'ora.crsd' on 't24db1' succeeded
    CRS-2673: Attempting to stop 'ora.asm' on 't24db1'
    CRS-2677: Stop of 'ora.asm' on 't24db1' succeeded
    CRS-2673: Attempting to stop 'ora.ctssd' on 't24db1'
    CRS-2677: Stop of 'ora.ctssd' on 't24db1' succeeded
    CRS-2673: Attempting to stop 'ora.cssdmonitor' on 't24db1'
    CRS-2677: Stop of 'ora.cssdmonitor' on 't24db1' succeeded
    CRS-2673: Attempting to stop 'ora.cssd' on 't24db1'
    CRS-2677: Stop of 'ora.cssd' on 't24db1' succeeded
    CRS-2673: Attempting to stop 'ora.gpnpd' on 't24db1'
    CRS-2677: Stop of 'ora.gpnpd' on 't24db1' succeeded
    CRS-2673: Attempting to stop 'ora.gipcd' on 't24db1'
    CRS-2677: Stop of 'ora.gipcd' on 't24db1' succeeded
    CRS-2673: Attempting to stop 'ora.mdnsd' on 't24db1'
    CRS-2677: Stop of 'ora.mdnsd' on 't24db1' succeeded
    CRS-2672: Attempting to start 'ora.mdnsd' on 't24db1'
    CRS-2676: Start of 'ora.mdnsd' on 't24db1' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 't24db1'
    CRS-2676: Start of 'ora.gipcd' on 't24db1' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 't24db1'
    CRS-2676: Start of 'ora.gpnpd' on 't24db1' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 't24db1'
    CRS-2676: Start of 'ora.cssdmonitor' on 't24db1' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 't24db1'
    CRS-2672: Attempting to start 'ora.diskmon' on 't24db1'
    CRS-2676: Start of 'ora.diskmon' on 't24db1' succeeded
    CRS-2676: Start of 'ora.cssd' on 't24db1' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 't24db1'
    CRS-2676: Start of 'ora.ctssd' on 't24db1' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 't24db1'
    CRS-2676: Start of 'ora.asm' on 't24db1' succeeded
    CRS-2672: Attempting to start 'ora.crsd' on 't24db1'
    CRS-2676: Start of 'ora.crsd' on 't24db1' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 't24db1'
    CRS-2676: Start of 'ora.evmd' on 't24db1' succeeded
    Timed out waiting for the CRS stack to start.
    I can also do the ocrcheck and i am also able to connect to ASM.
    crs process is not coming up.
    I tried to start it manually but no luck.
    alert log
    [crsd(11534476)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:29:50.913
    [crsd(11534476)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:29:52.130
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:29:53.760
    [crsd(11468998)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:29:54.029
    [crsd(11468998)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:29:55.246
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:29:55.246
    [ohasd(6947036)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    2011-09-21 20:51:59.589
    [crsd(8650916)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:51:59.854
    [crsd(8650916)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:00.895
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:02.588
    [crsd(9306182)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:52:02.852
    [crsd(9306182)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:04.024
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:05.693
    [crsd(9306184)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:52:05.961
    [crsd(9306184)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:07.174
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:08.807
    [crsd(11337884)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:52:09.078
    [crsd(11337884)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:10.289
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:11.942
    [crsd(11206904)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:52:12.207
    [crsd(11206904)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:13.400
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:15.031
    [crsd(11337894)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:52:15.302
    [crsd(11337894)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:16.521
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:18.132
    [crsd(11337896)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:52:18.403
    [crsd(11337896)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:19.627
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:21.225
    [crsd(9306200)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:52:21.498
    [crsd(9306200)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:22.717
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:25.895
    [crsd(8650934)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:52:26.176
    [crsd(8650934)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:27.369
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:29.046
    [crsd(9306206)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:52:29.316
    [crsd(9306206)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:30.511
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:32.169
    [crsd(11337908)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:52:32.435
    [crsd(11337908)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:52:33.629
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:52:33.630
    [ohasd(6947036)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    2011-09-21 20:57:11.777
    [ctssd(8388722)]CRS-2405:The Cluster Time Synchronization Service on host t24db1 is shutdown by user
    2011-09-21 20:58:15.374
    [ctssd(8519782)]CRS-2403:The Cluster Time Synchronization Service on host t24db1 is in observer mode.
    2011-09-21 20:58:15.391
    [ctssd(8519782)]CRS-2407:The new Cluster Time Synchronization Service reference node is host t24db1.
    2011-09-21 20:58:16.064
    [ctssd(8519782)]CRS-2401:The Cluster Time Synchronization Service started on host t24db1.
    2011-09-21 20:58:32.499
    [crsd(11862176)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:58:33.649
    [crsd(11862176)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:58:34.450
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:58:36.075
    [crsd(11534462)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:58:36.339
    [crsd(11534462)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:58:37.560
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:58:39.190
    [crsd(9699394)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:58:39.555
    [crsd(9699394)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:58:40.677
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:58:42.311
    [crsd(9699396)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:58:42.586
    [crsd(9699396)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:58:43.790
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:58:45.435
    [crsd(11600058)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:58:45.720
    [crsd(11600058)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:58:46.909
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:58:48.536
    [crsd(11600060)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:58:48.799
    [crsd(11600060)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:58:50.037
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:58:51.670
    [crsd(9699410)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:58:51.940
    [crsd(9699410)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:58:53.151
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:58:56.298
    [crsd(11600072)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:58:56.559
    [crsd(11600072)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:58:57.784
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:58:59.428
    [crsd(9699414)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:58:59.733
    [crsd(9699414)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:59:00.910
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:59:02.522
    [crsd(9699416)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:59:02.798
    [crsd(9699416)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:59:04.030
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:59:05.624
    [crsd(9699418)]CRS-1012:The OCR service started on node t24db1.
    2011-09-21 20:59:05.899
    [crsd(9699418)]CRS-1201:CRSD started on node t24db1.
    2011-09-21 20:59:07.120
    [ohasd(6947036)]CRS-2765:Resource 'ora.crsd' has failed on server 't24db1'.
    2011-09-21 20:59:07.121
    [ohasd(6947036)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    crsd log
    2011-09-21 20:59:06.013: [ default][1]clsu_get_private_ip_addresses: required buffer number is 2. Buffer number passed in 1 is not enough. Return [8]
    2011-09-21 20:59:06.013: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess (0)
    2011-09-21 20:59:06.014: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
    2011-09-21 20:59:06.015: [  CRSCCL][1]Listening endpoint created sucessfully @ (ADDRESS=(PROTOCOL=tcp)(DEV=52)(HOST=192.168.50.15)(PORT=49390)).con = 116b2c4f0
    2011-09-21 20:59:06.018: [  CRSCCL][10799]CSS Group Registration complete.
    2011-09-21 20:59:06.018: [  CRSCCL][10799]cclGetMemberData called
    2011-09-21 20:59:06.020: [  CRSCCL][10799]Obtained first membership map.
    2011-09-21 20:59:06.020: [  CRSCCL][10799]Dumping member data ------------------
    2011-09-21 20:59:06.020: [  CRSCCL][10799]Member (1, 1411327295) on node port=.
    2011-09-21 20:59:06.020: [  CRSCCL][10799]Done ------------------
    2011-09-21 20:59:06.020: [  CRSCCL][10799]Waiting for reconfigs
    2011-09-21 20:59:06.020: [  CRSCCL][11056]cclCommunicationHandler started.
    2011-09-21 20:59:06.020: [  CRSCCL][1]cclLibShutdown called
    2011-09-21 20:59:06.120: [  CRSCCL][10799]CCL Shutting down.
    2011-09-21 20:59:06.122: [  CRSCCL][10799]CSS Group Unregister complete.
    2011-09-21 20:59:06.122: [  CRSCCL][11056]Comunications Shutdown.
    2011-09-21 20:59:06.123: [  CRSCCL][10799]Membership Monitor exiting ...
    2011-09-21 20:59:06.123: [  CRSCCL][1]Clsc shutting down
    2011-09-21 20:59:06.123: [  CRSCCL][1]Clsc server shutdown
    2011-09-21 20:59:06.123: [ COMMCRS][1]clscugblmterm: (1158d7970) cleaning up icon (116b2bef0) with 1 cons
    2011-09-21 20:59:06.123: [ COMMCRS][1]clscugblmterm: (1158d7970) cleaning up open icon (116b2bef0)
    2011-09-21 20:59:06.123: [  CRSCCL][1]Clsc UGBLM shutdown
    2011-09-21 20:59:06.123: [  CRSCCL][1]Clsc shutdown Done
    2011-09-21 20:59:06.123: [  CRSCCL][1]ccllibShutdown done.
    2011-09-21 20:59:06.123: [CLSFRAME][1] clsCclInit returned failure:4
    2011-09-21 20:59:06.123: [CLSFRAME][1] Unable to start module-to-module comms: 2
    2011-09-21 20:59:06.123: [    CRSD][1][PANIC] CRSD exiting: unable to start CLS framework
    2011-09-21 20:59:06.123: [    CRSD][1] Done.
    Please assist me on this.
    Thanks and Regards,
    Daniesh

    1. try to give root capabilities with CAP_NUMA_ATTACH, CAP_BYPASS_RAC_VMM, and CAP_PROPAGATE
    2. install 11.2.0.2 version
    I solved the problem rootsh fails to bring up crs.

  • Root.sh fails, sles10 sp3 64-bit, 11gr2

    When i run the root.sh on first node, it fails.
    In the rootcrs_node1.log file i can see bellow entries
    2010-05-08 11:36:15: Executing as oracle: /u01/grid/11gr2/bin/asmca -silent -diskGroupName GRID_DG -diskList ORCL:ASMDISK1,ORCL:ASMDISK2,ORCL:ASMDISK3,ORCL:ASMDISK4 -redundancy EXTERNAL -configureLocalASM
    2010-05-08 11:36:15: Running as user oracle: /u01/grid/11gr2/bin/asmca -silent -diskGroupName GRID_DG -diskList ORCL:ASMDISK1,ORCL:ASMDISK2,ORCL:ASMDISK3,ORCL:ASMDISK4 -redundancy EXTERNAL -configureLocalASM
    2010-05-08 11:36:15: Invoking "/u01/grid/11gr2/bin/asmca -silent -diskGroupName GRID_DG -diskList ORCL:ASMDISK1,ORCL:ASMDISK2,ORCL:ASMDISK3,ORCL:ASMDISK4 -redundancy EXTERNAL -configureLocalASM" as user "oracle"
    2010-05-08 11:36:32: Configuration of ASM failed, see logs for details
    2010-05-08 11:36:32: Did not succssfully configure and start ASM
    2010-05-08 11:36:32: Exiting exclusive mode
    I am using device mapper multipath with oracleasm, the ORACLEASM_SCANORDER="dm" and ORACLEASM_SCANEXCLUDE="sd" set in /etc/sysconfig/oracleasm file.
    when i run the root.sh, the bellow will be displayed
    [main] [ 2010-05-08 11:36:16.871 IST ] [UsmcaCmdLineParser.discoverRAC:222] Cluster Existence Check
    [main] [ 2010-05-08 11:36:16.907 IST ] [Version.isPre:274] version to be checked 11.2.0.1.0 major version to check against10
    [main] [ 2010-05-08 11:36:16.907 IST ] [Version.isPre:285] isPre.java: Returning FALSE
    [main] [ 2010-05-08 11:36:16.912 IST ] [OCR.loadLibrary:308]
    Inside constructor of OCR
    [main] [ 2010-05-08 11:36:16.933 IST ] [OCR.loadLibrary:316] Going to load the ocr library
    [main] [ 2010-05-08 11:36:16.934 IST ] [sPlatform.isHybrid:66] osName=Linux osArch=amd64 JVM=64 rc=false
    [main] [ 2010-05-08 11:36:16.935 IST ] [sPlatform.isHybrid:66] osName=Linux osArch=amd64 JVM=64 rc=false
    [main] [ 2010-05-08 11:36:16.936 IST ] [Library.load:254] Loading library /u01/grid/11gr2/lib/libsrvmocr11.so
    [main] [ 2010-05-08 11:36:16.951 IST ] [OCR.loadLibrary:318] loaded ocr libraries
    [main] [ 2010-05-08 11:36:16.952 IST ] [OCR.isCluster:939] Calling OCRNative for isCluster()
    [main] [ 2010-05-08 11:36:16.956 IST ] [OCR.isCluster:943] OCR Result status = true
    [main] [ 2010-05-08 11:36:16.956 IST ] [OCR.isCluster:955] Bolean result = true
    [main] [ 2010-05-08 11:36:16.962 IST ] [HASContext.<init>:100] moduleInit = 7
    [main] [ 2010-05-08 11:36:16.962 IST ] [sPlatform.isHybrid:66] osName=Linux osArch=amd64 JVM=64 rc=false
    [main] [ 2010-05-08 11:36:16.963 IST ] [sPlatform.isHybrid:66] osName=Linux osArch=amd64 JVM=64 rc=false
    [main] [ 2010-05-08 11:36:16.964 IST ] [Library.load:254] Loading library /u01/grid/11gr2/lib/libsrvmhas11.so
    [main] [ 2010-05-08 11:36:16.968 IST ] [Util.<init>:86] Util Instance created.
    [main] [ 2010-05-08 11:36:16.979 IST ] [ClusterAlias.<init>:86] ClusterAlias Instance created.
    [main] [ 2010-05-08 11:36:17.030 IST ] [UsmcaCmdLineParser.discoverRAC:226] cluster existence:true
    [main] [ 2010-05-08 11:36:17.039 IST ] [OracleHome.getNodeNames:311] inside getNodeNames
    [main] [ 2010-05-08 11:36:17.408 IST ] [InventoryUtil.getOUIInvSession:347] setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS
    [main] [ 2010-05-08 11:36:17.415 IST ] [OracleHome.isClusterInstalled:293] bClusterInstalled=true
    [main] [ 2010-05-08 11:36:17.415 IST ] [UsmcaCmdLineParser.discoverRAC:240] Cluster flag true
    [main] [ 2010-05-08 11:36:23.946 IST ] [NetworkUtils.checkListenerStatus:420] Checking default listener status
    [main] [ 2010-05-08 11:36:23.947 IST ] [NetworkUtils.checkListenerStatus:425] Checking listener status: LISTENER
    [main] [ 2010-05-08 11:36:24.000 IST ] [NetworkUtils.checkListenerStatus:431] lsnrct status output
    LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 08-MAY-2010 11:36:23
    Copyright (c) 1991, 2009, Oracle. All rights reserved.
    Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
    TNS-12541: TNS:no listener
    TNS-12560: TNS:protocol adapter error
    TNS-00511: No listener
    Linux Error: 111: Connection refused
    [main] [ 2010-05-08 11:36:24.160 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: null
    [main] [ 2010-05-08 11:36:24.211 IST ] [InventoryUtil.getOUIInvSession:347] setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS
    [main] [ 2010-05-08 11:36:24.211 IST ] [OracleHome.getVersion:896] Homeinfo /u01/grid/11gr2,1
    [main] [ 2010-05-08 11:36:24.556 IST ] [OracleHome.getVersion:943] Current Version From Inventory: null
    [main] [ 2010-05-08 11:36:24.556 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:24.557 IST ] [OracleHome.getVersion:981] adding oracle home to sqlplus env
    [main] [ 2010-05-08 11:36:24.784 IST ] [OracleHome.getVersion:988] /u01/grid/11gr2/bin/sqlplus Banner:
    SQL*Plus: Release 11.2.0.1.0 Production
    [main] [ 2010-05-08 11:36:24.784 IST ] [OracleHome.getVersion:1006] Current version from sqlplus: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:24.784 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:24.784 IST ] [OracleHome.getVersion:943] Current Version From Inventory: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:24.785 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:24.786 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:24.786 IST ] [OracleHome.getVersion:943] Current Version From Inventory: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:24.787 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:24.787 IST ] [SQLPlusEngine.getCmmdParams:222] m_home 11.2.0.1.0
    [main] [ 2010-05-08 11:36:24.787 IST ] [SQLPlusEngine.getCmmdParams:223] version > 112 true
    [main] [ 2010-05-08 11:36:24.788 IST ] [SQLEngine.getEnvParams:547] Default NLS_LANG: AMERICAN_AMERICA.AL32UTF8
    [main] [ 2010-05-08 11:36:24.788 IST ] [SQLEngine.getEnvParams:557] NLS_LANG: AMERICAN_AMERICA.AL32UTF8
    [main] [ 2010-05-08 11:36:24.789 IST ] [SQLEngine.initialize:317] Execing SQLPLUS/SVRMGR process...
    [main] [ 2010-05-08 11:36:24.794 IST ] [SQLEngine.initialize:354] m_bReaderStarted: false
    [main] [ 2010-05-08 11:36:24.794 IST ] [SQLEngine.initialize:358] Starting Reader Thread...
    [main] [ 2010-05-08 11:36:24.807 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: null
    [main] [ 2010-05-08 11:36:24.810 IST ] [InventoryUtil.getOUIInvSession:347] setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS
    [main] [ 2010-05-08 11:36:24.811 IST ] [OracleHome.getVersion:896] Homeinfo /u01/grid/11gr2,1
    [main] [ 2010-05-08 11:36:24.967 IST ] [OracleHome.getVersion:943] Current Version From Inventory: null
    [main] [ 2010-05-08 11:36:24.967 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:24.968 IST ] [OracleHome.getVersion:981] adding oracle home to sqlplus env
    [main] [ 2010-05-08 11:36:24.988 IST ] [OracleHome.getVersion:988] /u01/grid/11gr2/bin/sqlplus Banner:
    SQL*Plus: Release 11.2.0.1.0 Production
    [main] [ 2010-05-08 11:36:24.989 IST ] [OracleHome.getVersion:1006] Current version from sqlplus: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:29.761 IST ] [KfodUtil.kfodOutput:232] /u01/grid/11gr2/bin/kfod
    [main] [ 2010-05-08 11:36:29.761 IST ] [KfodUtil.kfodOutput:232] nohdr=TRUE
    [main] [ 2010-05-08 11:36:29.762 IST ] [KfodUtil.kfodOutput:232] OP=DISKS DISK=ASM DS=TRUE
    [main] [ 2010-05-08 11:36:29.762 IST ] [KfodUtil.kfodOutput:232] status=true
    [main] [ 2010-05-08 11:36:30.801 IST ] [ClusterUtil.<init>:65] ClusterUtil Instance created
    [main] [ 2010-05-08 11:36:30.802 IST ] [ClusterUtil.<init>:68] ClusterUtil m_bSKGXN=true
    [main] [ 2010-05-08 11:36:30.803 IST ] [ClusterUtil.<init>:65] ClusterUtil Instance created
    [main] [ 2010-05-08 11:36:30.803 IST ] [ClusterUtil.<init>:68] ClusterUtil m_bSKGXN=true
    [main] [ 2010-05-08 11:36:30.803 IST ] [ClusterUtil.getCSSMode:463] Returned from native code
    [main] [ 2010-05-08 11:36:30.804 IST ] [ClusterUtil.getCSSMode:465] Returned from native code
    [main] [ 2010-05-08 11:36:30.806 IST ] [USMInstance.createASMInitFile:3227] creating init file /u01/grid/11gr2/dbs/init+ASM1.ora
    [main] [ 2010-05-08 11:36:30.810 IST ] [InitParamAttributes.sortParams:3774] m_sortOn:-1 sortOn:4
    [main] [ 2010-05-08 11:36:30.840 IST ] [InitParamAttributes.createInitFile:1808] Processing init param asm_power_limit
    [main] [ 2010-05-08 11:36:30.841 IST ] [InitParamAttributes.createInitFile:1813] The value is 1
    [main] [ 2010-05-08 11:36:30.841 IST ] [InitParamAttributes.createInitFile:1808] Processing init param diagnostic_dest
    [main] [ 2010-05-08 11:36:30.841 IST ] [InitParamAttributes.createInitFile:1813] The value is /etc/oracle
    [main] [ 2010-05-08 11:36:30.842 IST ] [InitParamAttributes.createInitFile:1808] Processing init param instance_type
    [main] [ 2010-05-08 11:36:30.842 IST ] [InitParamAttributes.createInitFile:1813] The value is asm
    [main] [ 2010-05-08 11:36:30.842 IST ] [InitParamAttributes.createInitFile:1808] Processing init param large_pool_size
    [main] [ 2010-05-08 11:36:30.842 IST ] [InitParamAttributes.createInitFile:1813] The value is 12M
    [main] [ 2010-05-08 11:36:30.843 IST ] [InitParamAttributes.createInitFile:1808] Processing init param remote_login_passwordfile
    [main] [ 2010-05-08 11:36:30.843 IST ] [InitParamAttributes.createInitFile:1813] The value is EXCLUSIVE
    [main] [ 2010-05-08 11:36:30.845 IST ] [SQLEngine.done:2148] Done called
    [main] [ 2010-05-08 11:36:30.846 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: null
    [main] [ 2010-05-08 11:36:30.849 IST ] [InventoryUtil.getOUIInvSession:347] setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS
    [main] [ 2010-05-08 11:36:30.849 IST ] [OracleHome.getVersion:896] Homeinfo /u01/grid/11gr2,1
    [main] [ 2010-05-08 11:36:30.995 IST ] [OracleHome.getVersion:943] Current Version From Inventory: null
    [main] [ 2010-05-08 11:36:30.995 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:30.996 IST ] [OracleHome.getVersion:981] adding oracle home to sqlplus env
    [main] [ 2010-05-08 11:36:31.130 IST ] [OracleHome.getVersion:988] /u01/grid/11gr2/bin/sqlplus Banner:
    SQL*Plus: Release 11.2.0.1.0 Production
    [main] [ 2010-05-08 11:36:31.131 IST ] [OracleHome.getVersion:1006] Current version from sqlplus: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:31.131 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:31.131 IST ] [OracleHome.getVersion:943] Current Version From Inventory: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:31.131 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:31.132 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:31.132 IST ] [OracleHome.getVersion:943] Current Version From Inventory: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:31.132 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:31.133 IST ] [SQLPlusEngine.getCmmdParams:222] m_home 11.2.0.1.0
    [main] [ 2010-05-08 11:36:31.133 IST ] [SQLPlusEngine.getCmmdParams:223] version > 112 true
    [main] [ 2010-05-08 11:36:31.133 IST ] [SQLEngine.getEnvParams:547] Default NLS_LANG: AMERICAN_AMERICA.AL32UTF8
    [main] [ 2010-05-08 11:36:31.134 IST ] [SQLEngine.getEnvParams:557] NLS_LANG: AMERICAN_AMERICA.AL32UTF8
    [main] [ 2010-05-08 11:36:31.135 IST ] [SQLEngine.initialize:317] Execing SQLPLUS/SVRMGR process...
    [main] [ 2010-05-08 11:36:31.142 IST ] [SQLEngine.initialize:354] m_bReaderStarted: false
    [main] [ 2010-05-08 11:36:31.142 IST ] [SQLEngine.initialize:358] Starting Reader Thread...
    [main] [ 2010-05-08 11:36:31.160 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: null
    [main] [ 2010-05-08 11:36:31.163 IST ] [InventoryUtil.getOUIInvSession:347] setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS
    [main] [ 2010-05-08 11:36:31.163 IST ] [OracleHome.getVersion:896] Homeinfo /u01/grid/11gr2,1
    [main] [ 2010-05-08 11:36:31.276 IST ] [OracleHome.getVersion:943] Current Version From Inventory: null
    [main] [ 2010-05-08 11:36:31.276 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:31.277 IST ] [OracleHome.getVersion:981] adding oracle home to sqlplus env
    [main] [ 2010-05-08 11:36:31.298 IST ] [OracleHome.getVersion:988] /u01/grid/11gr2/bin/sqlplus Banner:
    SQL*Plus: Release 11.2.0.1.0 Production
    [main] [ 2010-05-08 11:36:31.298 IST ] [OracleHome.getVersion:1006] Current version from sqlplus: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:32.044 IST ] [SQLEngine.done:2148] Done called
    [main] [ 2010-05-08 11:36:32.045 IST ] [USMInstance.configureLocalASM:2743] ORA-48141: error creating directory during ADR initialization [etc/oracle/diag]
    ORA-48189: OS command to create directory failed
    [main] [ 2010-05-08 11:36:32.045 IST ] [SQLEngine.reInitialize:730] Reinitializing SQLEngine...
    [main] [ 2010-05-08 11:36:32.045 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:32.046 IST ] [OracleHome.getVersion:943] Current Version From Inventory: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:32.046 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:32.046 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:32.046 IST ] [OracleHome.getVersion:943] Current Version From Inventory: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:32.047 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:32.047 IST ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:32.047 IST ] [OracleHome.getVersion:943] Current Version From Inventory: 11.2.0.1.0
    [main] [ 2010-05-08 11:36:32.048 IST ] [OracleHome.getVersion:948] using sqlplus: /u01/grid/11gr2/bin/sqlplus
    [main] [ 2010-05-08 11:36:32.048 IST ] [SQLPlusEngine.getCmmdParams:222] m_home 11.2.0.1.0
    [main] [ 2010-05-08 11:36:32.048 IST ] [SQLPlusEngine.getCmmdParams:223] version > 112 true
    [main] [ 2010-05-08 11:36:32.049 IST ] [SQLEngine.getEnvParams:547] Default NLS_LANG: AMERICAN_AMERICA.AL32UTF8
    [main] [ 2010-05-08 11:36:32.049 IST ] [SQLEngine.getEnvParams:557] NLS_LANG: AMERICAN_AMERICA.AL32UTF8
    [main] [ 2010-05-08 11:36:32.050 IST ] [SQLEngine.initialize:317] Execing SQLPLUS/SVRMGR process...
    [main] [ 2010-05-08 11:36:32.057 IST ] [SQLEngine.initialize:354] m_bReaderStarted: false
    [main] [ 2010-05-08 11:36:32.057 IST ] [SQLEngine.initialize:358] Starting Reader Thread...
    [main] [ 2010-05-08 11:36:32.058 IST ] [SQLEngine.initialize:407] Waiting for m_bReaderStarted to be true
    [main] [ 2010-05-08 11:36:32.094 IST ] [SQLEngine.done:2148] Done called
    [main] [ 2010-05-08 11:36:32.111 IST ] [OsUtilsBase.deleteFile:1711] OsUtilsBase.deleteFile: /u01/grid/11gr2/dbs/init+ASM1.ora
    ASM failed to start. Check /etc/oracle/cfgtoollogs/asmca/asmca-10050811AM3616.log for details.
    Configuration of ASM failed, see logs for details

    Hi,
    this:
    Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
    TNS-12541: TNS:no listener
    TNS-12560: TNS:protocol adapter error
    TNS-00511: No listener
    Linux Error: 111: Connection refused
    Could indicate a problem with identifiying the hostname correctly or a firewall problem.
    So either your hostname command does not return something correctly or your /etc/hosts contains errors or your firewall blocks the connection to the host.
    But since it does not contain a hostname, I guess a configuration error with the host.
    Sebastian

  • Run root.sh Failed to create or upgrade OLR (oracle11gr2+AIX6.1)

    2011-12-29 19:38:54: The configuration parameter file /oracle/grid/11.2/grid/crs/install/crsconfig_params is valid
    2011-12-29 19:38:54: Checking for super user privileges
    2011-12-29 19:38:54: User has super user privileges
    2011-12-29 19:38:54: ### Printing the configuration values from files:
    2011-12-29 19:38:54: /oracle/grid/11.2/grid/crs/install/crsconfig_params
    2011-12-29 19:38:54: /oracle/grid/11.2/grid/crs/install/s_crsconfig_defs
    2011-12-29 19:38:54: ASM_DISCOVERY_STRING=/dev/rup*
    2011-12-29 19:38:54: ASM_DISKS=/dev/rupdisk0,/dev/rupdisk1,/dev/rupdisk2
    2011-12-29 19:38:54: ASM_DISK_GROUP=CRS
    2011-12-29 19:38:54: ASM_REDUNDANCY=NORMAL
    2011-12-29 19:38:54: ASM_SPFILE=
    2011-12-29 19:38:54: ASM_UPGRADE=false
    2011-12-29 19:38:54: CLSCFG_MISSCOUNT=
    2011-12-29 19:38:54: CLUSTER_GUID=
    2011-12-29 19:38:54: CLUSTER_NAME=yhscluster
    2011-12-29 19:38:54: CRS_NODEVIPS="yhsscore1vip/255.255.255.192/en2,yhsscore2vip/255.255.255.192/en2"
    2011-12-29 19:38:54: CRS_STORAGE_OPTION=1
    2011-12-29 19:38:54: CSS_LEASEDURATION=400
    2011-12-29 19:38:54: DIRPREFIX=
    2011-12-29 19:38:54: DISABLE_OPROCD=0
    2011-12-29 19:38:54: EMBASEJAR_NAME=oemlt.jar
    2011-12-29 19:38:54: EWTJAR_NAME=ewt3.jar
    2011-12-29 19:38:54: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
    2011-12-29 19:38:54: GNS_ADDR_LIST=
    2011-12-29 19:38:54: GNS_ALLOW_NET_LIST=
    2011-12-29 19:38:54: GNS_CONF=false
    2011-12-29 19:38:54: GNS_DENY_ITF_LIST=
    2011-12-29 19:38:54: GNS_DENY_NET_LIST=
    2011-12-29 19:38:54: GNS_DOMAIN_LIST=
    2011-12-29 19:38:54: GPNPCONFIGDIR=/oracle/grid/11.2/grid
    2011-12-29 19:38:54: GPNPGCONFIGDIR=/oracle/grid/11.2/grid
    2011-12-29 19:38:54: GPNP_PA=
    2011-12-29 19:38:54: HELPJAR_NAME=help4.jar
    2011-12-29 19:38:54: HOST_NAME_LIST=yhsscore1,yhsscore2
    2011-12-29 19:38:54: ID=/etc
    2011-12-29 19:38:54: INIT=/usr/sbin/init
    2011-12-29 19:38:54: IT=/etc/inittab
    2011-12-29 19:38:54: JEWTJAR_NAME=jewt4.jar
    2011-12-29 19:38:54: JLIBDIR=/oracle/grid/11.2/grid/jlib
    2011-12-29 19:38:54: JREDIR=/oracle/grid/11.2/grid/jdk/jre/
    2011-12-29 19:38:54: LANGUAGE_ID=AMERICAN_AMERICA.WE8ISO8859P1
    2011-12-29 19:38:54: MSGFILE=/var/adm/messages
    2011-12-29 19:38:54: NETCFGJAR_NAME=netcfg.jar
    2011-12-29 19:38:54: NETWORKS="en2"/53.2.1.0:public,"en3"/10.0.0.0:cluster_interconnect
    2011-12-29 19:38:54: NEW_HOST_NAME_LIST=
    2011-12-29 19:38:54: NEW_NODEVIPS="yhsscore1vip/255.255.255.192/en2,yhsscore2vip/255.255.255.192/en2"
    2011-12-29 19:38:54: NEW_NODE_NAME_LIST=
    2011-12-29 19:38:54: NEW_PRIVATE_NAME_LIST=
    2011-12-29 19:38:54: NODELIST=yhsscore1,yhsscore2
    2011-12-29 19:38:54: NODE_NAME_LIST=yhsscore1,yhsscore2
    2011-12-29 19:38:54: OCFS_CONFIG=
    2011-12-29 19:38:54: OCRCONFIG=/etc/oracle/ocr.loc
    2011-12-29 19:38:54: OCRCONFIGDIR=/etc/oracle
    2011-12-29 19:38:54: OCRID=
    2011-12-29 19:38:54: OCRLOC=ocr.loc
    2011-12-29 19:38:54: OCR_LOCATIONS=NO_VAL
    2011-12-29 19:38:54: OLASTGASPDIR=/etc/oracle/lastgasp
    2011-12-29 19:38:54: OLD_CRS_HOME=
    2011-12-29 19:38:54: OLRCONFIG=/etc/oracle/olr.loc
    2011-12-29 19:38:54: OLRCONFIGDIR=/etc/oracle
    2011-12-29 19:38:54: OLRLOC=olr.loc
    2011-12-29 19:38:54: OPROCDCHECKDIR=/etc/oracle/oprocd/check
    2011-12-29 19:38:54: OPROCDDIR=/etc/oracle/oprocd
    2011-12-29 19:38:54: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
    2011-12-29 19:38:54: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
    2011-12-29 19:38:54: ORACLE_BASE=/oracle/grid/app/grid
    2011-12-29 19:38:54: ORACLE_HOME=/oracle/grid/11.2/grid
    2011-12-29 19:38:54: ORACLE_OWNER=grid
    2011-12-29 19:38:54: ORA_ASM_GROUP=dba
    2011-12-29 19:38:54: ORA_DBA_GROUP=dba
    2011-12-29 19:38:54: PRIVATE_NAME_LIST=
    2011-12-29 19:38:54: RCALLDIR=/etc/rc.d/rc2.d
    2011-12-29 19:38:54: RCKDIR=/etc/rc.d/rc2.d
    2011-12-29 19:38:54: RCSDIR=/etc/rc.d/rc2.d
    2011-12-29 19:38:54: RC_KILL=K19
    2011-12-29 19:38:54: RC_KILL_OLD=S96
    2011-12-29 19:38:54: RC_START=S96
    2011-12-29 19:38:54: SCAN_NAME=yhsscan
    2011-12-29 19:38:54: SCAN_PORT=1521
    2011-12-29 19:38:54: SCRBASE=/etc/oracle/scls_scr
    2011-12-29 19:38:54: SHAREJAR_NAME=share.jar
    2011-12-29 19:38:54: SILENT=false
    2011-12-29 19:38:54: SO_EXT=so
    2011-12-29 19:38:54: SRVCFGLOC=srvConfig.loc
    2011-12-29 19:38:54: SRVCONFIG=/var/opt/oracle/srvConfig.loc
    2011-12-29 19:38:54: SRVCONFIGDIR=/var/opt/oracle
    2011-12-29 19:38:54: VNDR_CLUSTER=false
    2011-12-29 19:38:54: VOTING_DISKS=NO_VAL
    2011-12-29 19:38:54: ### Printing other configuration values ###
    2011-12-29 19:38:54: CLSCFG_EXTRA_PARMS=
    2011-12-29 19:38:54: CRSDelete=0
    2011-12-29 19:38:54: CRSPatch=0
    2011-12-29 19:38:54: DEBUG=
    2011-12-29 19:38:54: DOWNGRADE=
    2011-12-29 19:38:54: HAS_GROUP=dba
    2011-12-29 19:38:54: HAS_USER=root
    2011-12-29 19:38:54: HOST=yhsscore1
    2011-12-29 19:38:54: IS_SIHA=0
    2011-12-29 19:38:54: OLR_DIRECTORY=/oracle/grid/11.2/grid/cdata
    2011-12-29 19:38:54: OLR_LOCATION=/oracle/grid/11.2/grid/cdata/yhsscore1.olr
    2011-12-29 19:38:54: ORA_CRS_HOME=/oracle/grid/11.2/grid
    2011-12-29 19:38:54: REMOTENODE=
    2011-12-29 19:38:54: SUPERUSER=root
    2011-12-29 19:38:54: UPGRADE=
    2011-12-29 19:38:54: VF_DISCOVERY_STRING=
    2011-12-29 19:38:54: addfile=/oracle/grid/11.2/grid/crs/install/crsconfig_addparams
    2011-12-29 19:38:54: crscfg_trace=1
    2011-12-29 19:38:54: crscfg_trace_file=/oracle/grid/11.2/grid/cfgtoollogs/crsconfig/rootcrs_yhsscore1.log
    2011-12-29 19:38:54: hosts=
    2011-12-29 19:38:54: oldcrshome=
    2011-12-29 19:38:54: oldcrsver=
    2011-12-29 19:38:54: osdfile=/oracle/grid/11.2/grid/crs/install/s_crsconfig_defs
    2011-12-29 19:38:54: parameters_valid=1
    2011-12-29 19:38:54: paramfile=/oracle/grid/11.2/grid/crs/install/crsconfig_params
    2011-12-29 19:38:54: platform_family=unix
    2011-12-29 19:38:54: srvctl_trc_suff=0
    2011-12-29 19:38:54: unlock_crshome=
    2011-12-29 19:38:54: user_is_superuser=1
    2011-12-29 19:38:54: ### Printing of configuration values complete ###
    2011-12-29 19:38:54: Oracle CRS stack is not configured yet
    2011-12-29 19:38:54: CRS is not yet configured. Hence, will proceed to configure CRS
    2011-12-29 19:38:54: Cluster-wide one-time actions... Done!
    2011-12-29 19:38:56: Oracle CRS home = /oracle/grid/11.2/grid
    2011-12-29 19:38:56: Host name = yhsscore1
    2011-12-29 19:38:56: CRS user = grid
    2011-12-29 19:38:56: Oracle CRS home = /oracle/grid/11.2/grid
    2011-12-29 19:38:56: GPnP host = yhsscore1
    2011-12-29 19:38:56: Oracle GPnP home = /oracle/grid/11.2/grid/gpnp
    2011-12-29 19:38:56: Oracle GPnP local home = /oracle/grid/11.2/grid/gpnp/yhsscore1
    2011-12-29 19:38:56: GPnP directories verified.
    2011-12-29 19:38:56: Checking to see if Oracle CRS stack is already configured
    2011-12-29 19:38:56: Oracle CRS stack is not configured yet
    2011-12-29 19:38:56: ---Checking local gpnp setup...
    2011-12-29 19:38:56: The setup file "/oracle/grid/11.2/grid/gpnp/yhsscore1/profiles/peer/profile.xml" does not exist
    2011-12-29 19:38:56: The setup file "/oracle/grid/11.2/grid/gpnp/yhsscore1/wallets/peer/cwallet.sso" does not exist
    2011-12-29 19:38:56: The setup file "/oracle/grid/11.2/grid/gpnp/yhsscore1/wallets/prdr/cwallet.sso" does not exist
    2011-12-29 19:38:56: chk gpnphome /oracle/grid/11.2/grid/gpnp/yhsscore1: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
    2011-12-29 19:38:56: chk gpnphome /oracle/grid/11.2/grid/gpnp/yhsscore1: INVALID (bad profile/wallet)
    2011-12-29 19:38:56: ---Checking cluster-wide gpnp setup...
    2011-12-29 19:38:56: The setup file "/oracle/grid/11.2/grid/gpnp/profiles/peer/profile.xml" does not exist
    2011-12-29 19:38:56: The setup file "/oracle/grid/11.2/grid/gpnp/wallets/peer/cwallet.sso" does not exist
    2011-12-29 19:38:56: The setup file "/oracle/grid/11.2/grid/gpnp/wallets/prdr/cwallet.sso" does not exist
    2011-12-29 19:38:56: chk gpnphome /oracle/grid/11.2/grid/gpnp: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
    2011-12-29 19:38:56: chk gpnphome /oracle/grid/11.2/grid/gpnp: INVALID (bad profile/wallet)
    2011-12-29 19:38:56: gpnp setup checked: local valid? 0 cluster-wide valid? 0
    2011-12-29 19:38:56: gpnp setup: NONE
    2011-12-29 19:38:56: GPNP configuration required
    2011-12-29 19:38:56: Validating for SI-CSS configuration
    2011-12-29 19:38:56: Retrieving OCR main disk location
    2011-12-29 19:38:56: Opening file OCRCONFIG
    2011-12-29 19:38:56: Value () is set for key=ocrconfig_loc
    2011-12-29 19:38:56: Unable to retrieve ocr disk info
    2011-12-29 19:38:56: Checking to see if any 9i GSD is up
    2011-12-29 19:38:56: libskgxnBase_lib = /etc/ORCLcluster/oracm/lib/libskgxn2.so
    2011-12-29 19:38:56: libskgxn_lib = /opt/ORCLcluster/lib/libskgxn2.so
    2011-12-29 19:38:56: SKGXN library file does not exists
    2011-12-29 19:38:56: OLR location = /oracle/grid/11.2/grid/cdata/yhsscore1.olr
    2011-12-29 19:38:56: Oracle CRS Home = /oracle/grid/11.2/grid
    2011-12-29 19:38:56: Validating /etc/oracle/olr.loc file for OLR location /oracle/grid/11.2/grid/cdata/yhsscore1.olr
    2011-12-29 19:38:56: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig
    2011-12-29 19:38:56: Oracle CRS home = /oracle/grid/11.2/grid
    2011-12-29 19:38:56: Oracle cluster name = yhscluster
    2011-12-29 19:38:56: OCR locations = +CRS
    2011-12-29 19:38:56: Validating OCR
    2011-12-29 19:38:56: Retrieving OCR location used by previous installations
    2011-12-29 19:38:56: Opening file OCRCONFIG
    2011-12-29 19:38:56: Value () is set for key=ocrconfig_loc
    2011-12-29 19:38:56: Opening file OCRCONFIG
    2011-12-29 19:38:56: Value () is set for key=ocrmirrorconfig_loc
    2011-12-29 19:38:56: Opening file OCRCONFIG
    2011-12-29 19:38:56: Value () is set for key=ocrconfig_loc3
    2011-12-29 19:38:56: Opening file OCRCONFIG
    2011-12-29 19:38:56: Value () is set for key=ocrconfig_loc4
    2011-12-29 19:38:56: Opening file OCRCONFIG
    2011-12-29 19:38:56: Value () is set for key=ocrconfig_loc5
    2011-12-29 19:38:56: Checking if OCR sync file exists
    2011-12-29 19:38:56: No need to sync OCR file
    2011-12-29 19:38:56: OCR_LOCATION=+CRS
    2011-12-29 19:38:56: OCR_MIRROR_LOCATION=
    2011-12-29 19:38:56: OCR_MIRROR_LOC3=
    2011-12-29 19:38:56: OCR_MIRROR_LOC4=
    2011-12-29 19:38:56: OCR_MIRROR_LOC5=
    2011-12-29 19:38:56: Current OCR location=
    2011-12-29 19:38:56: Current OCR mirror location=
    2011-12-29 19:38:56: Current OCR mirror loc3=
    2011-12-29 19:38:56: Current OCR mirror loc4=
    2011-12-29 19:38:56: Current OCR mirror loc5=
    2011-12-29 19:38:56: Verifying current OCR settings with user entered values
    2011-12-29 19:38:56: Setting OCR locations in /etc/oracle/ocr.loc
    2011-12-29 19:38:56: Validating OCR locations in /etc/oracle/ocr.loc
    2011-12-29 19:38:56: Checking for existence of /etc/oracle/ocr.loc
    2011-12-29 19:38:56: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig
    2011-12-29 19:38:56: Setting ocr location +CRS
    2011-12-29 19:38:56: User grid has the required capabilities to run CSSD in realtime mode
    *2011-12-29 19:38:56: Creating or upgrading Oracle Local Registry (OLR)*
    *2011-12-29 19:38:56: Command return code of 255 (65280) from command: /oracle/grid/11.2/grid/bin/ocrconfig -local -upgrade grid dba*
    *2011-12-29 19:38:56: /oracle/grid/11.2/grid/bin/ocrconfig -local -upgrade failed with error: 255*
    *2011-12-29 19:38:56: Failed to create or upgrade OLR*
    帖子经 905068编辑过

    refer:-
    Command return code of 255 (65280) during Grid Infrastructure Installation
    http://coskan.wordpress.com/2009/12/07/root-sh-failed-after-asm-disk-creation-for-11gr2-grid-infrastructure/

  • Run root.sh failed after install grid

    i installed grid,and run root.sh to configure the cluster,it took more than 12 hours,finally get the fail report
    anybody can help?
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME= /opt/oracle/app/11.2.0/grid
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2011-05-18 23:42:50: Parsing the host name
    2011-05-18 23:42:50: Checking for super user privileges
    2011-05-18 23:42:50: User has super user privileges
    Using configuration parameter file: /opt/oracle/app/11.2.0/grid/crs/install/crsc
    onfig_params
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    root wallet
    root wallet cert
    root cert export
    peer wallet
    profile reader wallet
    　pa wallet
    peer wallet keys
    pa wallet keys
    peer cert request
    pa cert request
    peer cert
    pa cert
    peer root cert TP
    profile reader root cert TP
    pa root cert TP
    peer pa cert TP
    pa peer cert TP
    profile reader pa cert TP
    profile reader peer cert TP
    peer user cert
    pa user cert
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    ADVM/ACFS is not supported on centos-release-5-6.el5.centos.1
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac1-pub'
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1-pub'
    CRS-2676: Start of 'ora.mdnsd' on 'rac1-pub' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'rac1-pub' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1-pub'
    CRS-2676: Start of 'ora.gpnpd' on 'rac1-pub' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1-pub'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac1-pub' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac1-pub'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac1-pub'
    CRS-2676: Start of 'ora.diskmon' on 'rac1-pub' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac1-pub' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac1-pub'
    CRS-2676: Start of 'ora.ctssd' on 'rac1-pub' succeeded
    ASM created and started successfully.
    DiskGroup MyDG created successfully.
    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    CRS-2672: Attempting to start 'ora.crsd' on 'rac1-pub'
    CRS-2676: Start of 'ora.crsd' on 'rac1-pub' succeeded
    CRS-4256: Updating the profile
    Successful addition of voting disk 83a7a20dc30f4f62bf5fc4e4a7a18567.
    Successful addition of voting disk a8df71f68dbf4f22bf09f68aaa04de47.
    Successful addition of voting disk 60cec6ca14524fdabf777568d967b5d4.
    Successfully replaced voting disk group with +MyDG.
    CRS-4256: Updating the profile
    Failed to update the profile with the new discovery string
    CRS-4000: Command Replace failed, or completed with errors.
    Command return code of 1 (256) from command: /opt/oracle/app/11.2.0/grid/bin/crsctl replace votedisk +MyDG
    Voting file add failed
    Unable to communicate with the Cluster Synchronization Services daemon.
    CRS-4639: Could not contact Oracle High Availability Services
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /opt/oracle/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
    Stop of resource "ora.crsd -init" failed
    Failed to stop CRSD
    CRS-4639: Could not contact Oracle High Availability Services
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /opt/oracle/app/11.2.0/grid/bin/crsctl stop resource ora.asm -init
    Stop of resource "ora.asm -init" failed
    Failed to stop ASM
    CRS-4639: Could not contact Oracle High Availability Services
    CRS-4000: Command Stop failed, or completed with errors.
    Command return code of 1 (256) from command: /opt/oracle/app/11.2.0/grid/bin/crsctl stop resource ora.ctssd -init
    Stop of resource "ora.ctssd -init" failed
    Failed to stop OCTSSD
    Initial cluster configuration failed. See /opt/oracle/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_rac1-pub.log for details
    =======================
    and infor from /opt/oracle/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_rac1-pub.log
    2011-05-18 23:43:17: Registering ohasd
    2011-05-18 23:43:17: init file = /opt/oracle/app/11.2.0/grid/crs/init/init.ohasd
    2011-05-18 23:43:17: Copying file /opt/oracle/app/11.2.0/grid/crs/init/init.ohasd to /etc/init.d directory
    2011-05-18 23:43:17: Setting init.ohasd permission in /etc/init.d directory
    2011-05-18 23:43:17: init file = /opt/oracle/app/11.2.0/grid/crs/init/ohasd
    2011-05-18 23:43:17: Copying file /opt/oracle/app/11.2.0/grid/crs/init/ohasd to /etc/init.d directory
    2011-05-18 23:43:17: Setting ohasd permission in /etc/init.d directory
    2011-05-18 23:43:17: Removing "/etc/rc.d/rc3.d/S96ohasd"
    2011-05-18 23:43:17: Removing file /etc/rc.d/rc3.d/S96ohasd
    2011-05-18 23:43:17: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd
    2011-05-18 23:43:17: Failed to remove file:
    2011-05-18 23:43:17: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
    2011-05-18 23:43:17: Removing "/etc/rc.d/rc5.d/S96ohasd"
    2011-05-18 23:43:17: Removing file /etc/rc.d/rc5.d/S96ohasd
    2011-05-18 23:43:17: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd
    2011-05-18 23:43:17: Failed to remove file:
    2011-05-18 23:43:17: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
    2011-05-18 23:43:17: Removing "/etc/rc.d/rc0.d/K19ohasd"
    2011-05-18 23:43:17: Removing file /etc/rc.d/rc0.d/K19ohasd
    2011-05-18 23:43:17: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd
    2011-05-18 23:43:17: Failed to remove file:
    2011-05-18 23:43:17: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd
    2011-05-18 23:43:17: Removing "/etc/rc.d/rc1.d/K19ohasd"
    2011-05-18 23:43:17: Removing file /etc/rc.d/rc1.d/K19ohasd
    2011-05-18 23:43:17: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd
    2011-05-18 23:43:17: Failed to remove file:
    2011-05-18 23:43:17: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd
    2011-05-18 23:43:17: Removing "/etc/rc.d/rc2.d/K19ohasd"
    2011-05-18 23:43:17: Removing file /etc/rc.d/rc2.d/K19ohasd
    2011-05-18 23:43:17: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd
    2011-05-18 23:43:17: Failed to remove file:
    2011-05-18 23:43:17: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd
    2011-05-18 23:43:17: Removing "/etc/rc.d/rc4.d/K19ohasd"
    2011-05-18 23:43:17: Removing file /etc/rc.d/rc4.d/K19ohasd
    2011-05-18 23:43:17: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd
    2011-05-18 23:43:17: Failed to remove file:
    2011-05-18 23:43:17: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd
    2011-05-18 23:43:17: Removing "/etc/rc.d/rc6.d/K19ohasd"
    2011-05-18 23:43:17: Removing file /etc/rc.d/rc6.d/K19ohasd
    2011-05-18 23:43:17: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
    2011-05-18 23:43:17: Failed to remove file:
    2011-05-18 23:43:17: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
    2011-05-18 23:43:17: The file ohasd has been successfully linked to the RC directories
    2011-05-18 23:43:17: Starting ohasd
    2011-05-18 23:43:17: itab entries=
    2011-05-18 23:43:22: Created backup /etc/inittab.no_crs
    2011-05-18 23:43:22: Appending to /etc/inittab.tmp:
    2011-05-18 23:43:22: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
    2011-05-18 23:43:22: Done updating /etc/inittab.tmp
    2011-05-18 23:43:22: Saved /etc/inittab.crs
    2011-05-18 23:43:22: Installed new /etc/inittab
    2011-05-18 23:43:27: ohasd is starting
    2011-05-18 23:43:27: Checking ohasd
    2011-05-18 23:43:27: ohasd started successfully
    2011-05-18 23:43:27: Creating CRS resources and dependencies
    2011-05-18 23:43:27: Configuring HASD
    2011-05-18 23:43:27: Registering type ora.daemon.type
    2011-05-18 23:43:28: Registering type ora.mdns.type
    2011-05-18 23:43:28: Registering type ora.gpnp.type
    2011-05-18 23:43:29: Registering type ora.gipc.type
    2011-05-18 23:43:29: Registering type ora.cssd.type
    2011-05-18 23:43:30: Registering type ora.cssdmonitor.type
    2011-05-18 23:43:30: Registering type ora.crs.type
    2011-05-18 23:43:31: Registering type ora.evm.type
    2011-05-18 23:43:31: Registering type ora.ctss.type
    2011-05-18 23:43:31: Registering type ora.asm.type
    2011-05-18 23:43:32: Registering type ora.drivers.acfs.type
    2011-05-18 23:43:32: Registering type ora.diskmon.type
    2011-05-18 23:43:35: ADVM/ACFS is not supported on centos-release-5-6.el5.centos.1
    2011-05-18 23:43:35: ADVM/ACFS is not configured
    2011-05-18 23:43:35: Successfully created CRS resources for cluster daemon and ASM
    2011-05-18 23:43:35: Checking if initial configuration has been performed
    2011-05-18 23:43:35: Starting CSS in exclusive mode
    2011-05-18 23:44:13: CRS-2672: Attempting to start 'ora.gipcd' on 'rac1-pub'
    2011-05-18 23:44:13: CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1-pub'
    2011-05-18 23:44:13: CRS-2676: Start of 'ora.mdnsd' on 'rac1-pub' succeeded
    2011-05-18 23:44:13: CRS-2676: Start of 'ora.gipcd' on 'rac1-pub' succeeded
    2011-05-18 23:44:13: CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1-pub'
    2011-05-18 23:44:13: CRS-2676: Start of 'ora.gpnpd' on 'rac1-pub' succeeded
    2011-05-18 23:44:13: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1-pub'
    2011-05-18 23:44:13: CRS-2676: Start of 'ora.cssdmonitor' on 'rac1-pub' succeeded
    2011-05-18 23:44:13: CRS-2672: Attempting to start 'ora.cssd' on 'rac1-pub'
    2011-05-18 23:44:13: CRS-2672: Attempting to start 'ora.diskmon' on 'rac1-pub'
    2011-05-18 23:44:13: CRS-2676: Start of 'ora.diskmon' on 'rac1-pub' succeeded
    2011-05-18 23:44:13: CRS-2676: Start of 'ora.cssd' on 'rac1-pub' succeeded
    2011-05-18 23:44:13: Querying for existing CSS voting disks
    2011-05-18 23:44:13: Performing initial configuration for cluster
    2011-05-18 23:44:14: Start of resource "ora.ctssd -init" Succeeded
    2011-05-18 23:44:15: Configuring ASM via ASMCA
    2011-05-18 23:44:15: Executing as grid: /opt/oracle/app/11.2.0/grid/bin/asmca -silent -diskGroupName MyDG -diskList /dev/oracleasm/disks/DISK1,/dev/oracleasm/disks/DISK2,/dev/oracleasm/disks/DISK3 -redundancy NORMAL -diskString '/dev/oracleasm/disks/*' -configureLocalASM
    2011-05-18 23:44:15: Running as user grid: /opt/oracle/app/11.2.0/grid/bin/asmca -silent -diskGroupName MyDG -diskList /dev/oracleasm/disks/DISK1,/dev/oracleasm/disks/DISK2,/dev/oracleasm/disks/DISK3 -redundancy NORMAL -diskString '/dev/oracleasm/disks/*' -configureLocalASM
    2011-05-18 23:44:15: Invoking "/opt/oracle/app/11.2.0/grid/bin/asmca -silent -diskGroupName MyDG -diskList /dev/oracleasm/disks/DISK1,/dev/oracleasm/disks/DISK2,/dev/oracleasm/disks/DISK3 -redundancy NORMAL -diskString '/dev/oracleasm/disks/*' -configureLocalASM" as user "grid"
    2011-05-18 23:47:27: Creating or upgrading OCR keys
    2011-05-18 23:47:46: OCR keys are successfully populated
    2011-05-18 23:47:46: Invalidating repository location for Oracle 9i deployments
    2011-05-18 23:47:46: Executing clscfg
    2011-05-18 23:47:53: Oracle Cluster Registry initialization completed
    2011-05-18 23:47:59: Start of resource "ora.crsd -init" Succeeded
    2011-05-18 23:47:59: Creating voting files
    2011-05-18 23:47:59: Creating voting files in ASM diskgroup MyDG
    2011-05-18 23:47:59: Executing crsctl replace votedisk +MyDG
    2011-05-19 11:04:19: Command return code of 1 (256) from command: /opt/oracle/app/11.2.0/grid/bin/crsctl replace votedisk +MyDG
    2011-05-19 11:20:24: Voting file add failed
    2011-05-19 11:54:33: Exiting exclusive mode
    2011-05-19 12:11:11: Command return code of 1 (256) from command: /opt/oracle/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
    2011-05-19 12:13:15: Stop of resource "ora.crsd -init" failed
    2011-05-19 12:15:12: Failed to stop CRSD
    2011-05-19 12:20:52: Command return code of 1 (256) from command: /opt/oracle/app/11.2.0/grid/bin/crsctl stop resource ora.asm -init
    2011-05-19 12:20:55: Stop of resource "ora.asm -init" failed
    2011-05-19 12:20:55: Failed to stop ASM
    2011-05-19 12:23:59: Command return code of 1 (256) from command: /opt/oracle/app/11.2.0/grid/bin/crsctl stop resource ora.ctssd -init
    2011-05-19 12:24:05: Stop of resource "ora.ctssd -init" failed
    2011-05-19 12:24:05: Failed to stop OCTSSD
    2011-05-19 12:36:49: Initial cluster configuration failed. See /opt/oracle/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_rac1-pub.log for details

    Hi,
    sounds like a security/firewall problem.
    Because if during the configuration something is not "acting" it's mostly because packages are not delivered/recieved and "hang" instead of delivering error messages.
    To make sure deactivate any firewall, and if this is linux deactivate SE Linux.
    Next time please specify version and OS you encounter the error. This would make it easier to help you...
    Regards
    Sebastian

Maybe you are looking for

  • Email with images

    Hi,    Our requirement is to send images as part of email and also the email content is HTML. When we send the email the system in SCOT shows the mail has been sent successfully. But the at customer site the exchange server is blocking .bmp files and

  • How to switch off orb.properties locally.

    Hi, I am using 1.3 J2SE client to run weblogic EJBs. This is working good with sun's JDK. I need to connect to Weblogic from old Silverstream, which is also using working 1.3 Sun's jre, but it has set up it's own ORB in orb.properties (JRE/lib direct

  • Entire song list showin up under movies

    when i go into my videos an movie folders it doesnt show any movies or vidoes because i dont have any on my ipod but what it does show is my entire song list....how do i get my songs out of the movie folder/section

  • Cloud Connector - Contact Creation in Eloqua

    I am building a new Cloud Connector. I have reviewed the following doc http://img.en25.com/Web/Eloqua/EloquaCloudConnectorSampleGuide.pdf In the above example, the doc assumes the email address already exists in Eloqua. The Cloud Connector is used to

  • Why do I continue to lose entries in the Notes app

    I use Notes on my iPhone to store and save important numbers that I often need to reference.  I do backup my phone with my iMac and I also use iCloud. However repeatedly the my list of Numbers disappears from my iPhone and I end up having to reinstal