Crs doesn't start on second node

Guys,
RAC on 2 nodes
Release 10.2.0.5.0
Solaris 10
There was a problem with the cable that enables connection for the interconnect, but the problem has been solved. One of the nodes was evicted and all resources were move to the other node. Once the problem was solved I tried to start the cluster that was evicted but to no success. when I run crs_stat -t I get the infamous CRS-0184.
I have checked the ocr and olsnodes; ocr seems to be fine and the second node is recognized as part of the cluster.
cluvfy comp ocr -n lenin,trotsky -verbose
Verifying OCR integrity
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations.
Uniqueness check for OCR device passed.
Checking the version of OCR...
OCR of correct Version "2" exists.
Checking data integrity of OCR...
Data integrity check for OCR passed.
OCR integrity check passed.
Verification of OCR integrity was successful.
oracle@trotsky > cluvfy comp nodereach -n lenin,trotsky -srcnode trotsky -verbose
Verifying node reachability
Checking node reachability...
Check: Node reachability from node "trotsky"
Destination Node Reachable?
lenin yes
trotsky yes
Result: Node reachability check passed from node "trotsky".
I have checked /var/adm/messages and crs and cssd log but I didn't see anything that stands out...
I have also tried to delete the content of /var/tmp/.oracle and restart crs but again to no success.
I have read in another thread in this forum that crs problems are either related to the interconnect or ocr/voting disks but as mentioned before they seem to be OK.
I'm running out of ideas, any suggestions?
One of the nodes now holds both vip addresses:
bge0:1: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
inet 192.168.191.184 netmask ffffff00 broadcast 192.168.191.255
bge0:2: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
inet 192.168.191.182 netmask ffffff00 broadcast 192.168.191.255
Do I need to manually reconfigure the interface do that is then held by the second node?
Thanks in advance for your help

Cheers for your input!
The results on the suggested cluvfy command is: passed on all checks with the exception of the daemon liveness (as expected).
Excerpts from the different logs:
alert.log
2010-11-19 13:12:35.033
[cssd(4928)]CRS-1605:CSSD voting file is online: /dev/rdsk/c1t500601604BA03AEAd0s5. Details in /u01/crs/10.2.0/crs_1/log/trotsky/cssd/ocssd.log.
2010-11-19 13:12:35.050
[cssd(4928)]CRS-1605:CSSD voting file is online: /dev/rdsk/c1t500601604BA03AEAd0s4. Details in /u01/crs/10.2.0/crs_1/log/trotsky/cssd/ocssd.log.
2010-11-19 13:12:35.062
[cssd(4928)]CRS-1605:CSSD voting file is online: /dev/rdsk/c1t500601604BA03AEAd0s6. Details in /u01/crs/10.2.0/crs_1/log/trotsky/cssd/ocssd.log.
cssd.log
[    CSSD]2010-11-19 13:16:47.059 [21] >WARNING: clssnmLocalJoinEvent: takeover aborted due to ALIVE node on Disk
[    CSSD]2010-11-19 13:16:47.059 [21] >WARNING: clssnmRcfgMgrThread: not possible to join the cluster. Please reboot the node.
[    CSSD]2010-11-19 13:16:47.059 [21] >WARNING: clssnmReconfigThread: state(1) clusterState(0) exit
I have tried rebooting the node but that did not help.
crsd.log
2010-11-19 13:53:49.652: [  CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2010-11-19 13:53:50.889: [ COMMCRS][1802]clsc_connect: (1009ac310) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_trotsky_))
2010-11-19 13:53:50.889: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
2010-11-19 13:53:50.890: [  CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2010-11-19 13:53:51.899: [    CRSD][1][PANIC] CRSD exiting: Could not init the CSS context
2010-11-19 13:53:51.899: [    CRSD][1] Done.
Does this help?

Similar Messages

  • Timed out waiting for the CRS stack to start on Second node.

    Hi,
    I am trying to setup 2 node 11gR2 RAC in vmware. I face the Timed out waiting for the CRS stack to start error on second node while running root.sh.I checked the cluster log files located in  /u01/app/11.2.0/grid/log/node1/alertnode2.log it shows as mentioned below.But when i logged into ASM instance and checked the diskgroup and it is in mount state on both nodes,So i am really confused why i ended up with error in 2nd node on running root.sh. Can anyone tell me how to correct this error or any other checking needs to be done?
    /u01/app/11.2.0/grid/log/node1/alertnode2.log
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:10.314
    [crsd(25309)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
    2014-01-21 21:40:11.217
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:12.422
    [crsd(25335)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
    2014-01-21 21:40:13.327
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:14.514
    [crsd(25356)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
    2014-01-21 21:40:15.439
    [ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
    2014-01-21 21:40:15.440
    [ohasd(23261)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    root.sh output
    [root@node2 ~]# /u01/app/11.2.0/grid/root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /u01/app/11.2.0/grid
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
    The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
    [n]: y
       Copying dbhome to /usr/local/bin ...
    The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
    [n]: y
       Copying oraenv to /usr/local/bin ...
    The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
    [n]: y
       Copying coraenv to /usr/local/bin ...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2014-01-21 21:37:55: Parsing the host name
    2014-01-21 21:37:55: Checking for super user privileges
    2014-01-21 21:37:55: User has super user privileges
    Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_p                   
    arams
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node 11gdb, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'node2'
    CRS-2676: Start of 'ora.mdnsd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'node2'
    CRS-2676: Start of 'ora.gipcd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'node2'
    CRS-2676: Start of 'ora.gpnpd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'node2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'node2'
    CRS-2676: Start of 'ora.diskmon' on 'node2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'node2'
    CRS-2676: Start of 'ora.ctssd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'node2'
    CRS-2676: Start of 'ora.drivers.acfs' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.asm' on 'node2'
    CRS-2676: Start of 'ora.asm' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.crsd' on 'node2'
    CRS-2676: Start of 'ora.crsd' on 'node2' succeeded
    CRS-2672: Attempting to start 'ora.evmd' on 'node2'
    CRS-2676: Start of 'ora.evmd' on 'node2' succeeded
    Timed out waiting for the CRS stack to start.
    Regards,
    007

    Hi,
    Are you trying to install on Vmware Workstation or Vmware Server?
    If you are using vmware server add below line in your .vmx file for both node
    scsi1.sharedBus = "VIRTUAL"
    First check which scsi serial number are you using as above I have selected scsi1 serial during disk addition, 

  • ONS failed to start on second node

    Hi,
    I have a problem with ons on 10g rac running on linux 5.3
    on node 1 it is running without problem but on second node i got this error
    2009-04-08 16:30:41.318: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the
    2009-04-08 16:30:41.319: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    2009-04-08 16:30:41.319: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    Number of onsconfiguration retrieved, numcfg = 2
    onscfg[0]
    {node = rac1, port = 6200}
    Adding remote host rac1:6200
    onscfg[1]
    {node = rac2, port = 6
    2009-04-08 16:30:41.319: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: 200}
    Adding remote host rac2:6200
    RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    RCV: Permission d
    2009-04-08 16:30:41.319: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: enied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the OPMN server loca
    2009-04-08 16:30:41.319: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: l port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    Number of onsconfiguration retrieved, numcfg = 2
    onscfg[0]
    {node = rac1, port = 6200}
    Adding remote host rac1:6200
    o
    2009-04-08 16:30:41.319: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: nscfg[1]
    {node = rac2, port = 6200}
    Adding remote host rac2:6200
    onsctl: ons failed to start
    2009-04-08 16:30:41.319: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/app/crs
    2009-04-08 16:30:41.319: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: cmd = /u01/app/crs/bin/racgeut -e USRORA_DEBUG=0 540 /u01/app/crs/bin/onsctl start
    2009-04-08 16:30:41.320: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: rc = 1, time = 2.580s
    2009-04-08 16:30:42.148: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the
    2009-04-08 16:30:42.150: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    2009-04-08 16:30:42.150: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: RCV: Permission denied
    Communication error with the OPMN server local port.
    Check the OPMN log files
    Number of onsconfiguration retrieved, numcfg = 2
    onscfg[0]
    {node = rac1, port = 6200}
    Adding remote host rac1:6200
    onscfg[1]
    {node = rac2, port = 6
    2009-04-08 16:30:42.150: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: 200}
    Adding remote host rac2:6200
    ons is not running ...
    2009-04-08 16:30:42.151: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/app/crs
    2009-04-08 16:30:42.151: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: cmd = /u01/app/crs/bin/racgeut -e USRORA_DEBUG=0 540 /u01/app/crs/bin/onsctl ping
    2009-04-08 16:30:42.151: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: rc = 1, time = 0.840s
    2009-04-08 16:30:42.153: [    RACG][3065611968] [16553][3065611968][ora.rac2.ons]: end for resource = ora.rac2.ons, action = start, status = 1, time = 3.620s
    2009-04-08 16:30:44.376: [    RACG][3066242752] [17061][3066242752][ora.rac2.ons]: onsctl: shutting down ons daemon ...
    Number of onsconfiguration retrieved, numcfg = 2
    onscfg[0]
    {node = rac1, port = 6200}
    Adding remote host rac1:6200
    onscfg[1]
    {node = rac2, port = 6200}
    Adding remote host rac2:6200
    Any idea how to fix this?
    Thanks

    check the output for crs_getperm for the resource from both nodes. If you could, post them here.
    Regards,
    Ganesh

  • Failure at final check of Oracle CRS stack.10  on the second node

    Hi,
    I am trying to install Oracle Clusterware 10.2.0.1.0 in VM machines (2 nodes config) in Linux (OEL5) using VMware Server (2.0). Everything went very well one the first node upto running the root.sh. Running root.sh ended with Failure at final check of Oracle CRS stack 10 error.
    RAC1 root.sh output
    [root@rac1 crs]# ./root.sh
    WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
    WARNING: directory '/u01/crs/oracle/product' is not owned by root
    WARNING: directory '/u01/crs/oracle' is not owned by root
    WARNING: directory '/u01/crs' is not owned by root
    WARNING: directory '/u01' is not owned by root
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up NS directories
    Oracle Cluster Registry configuration upgraded successfully
    WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
    WARNING: directory '/u01/crs/oracle/product' is not owned by root
    WARNING: directory '/u01/crs/oracle' is not owned by root
    WARNING: directory '/u01/crs' is not owned by root
    WARNING: directory '/u01' is not owned by root
    assigning default hostname rac1 for node 1.
    assigning default hostname rac2 for node 2.
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: rac1 rac1-priv rac1
    node 2: rac2 rac2-priv rac2
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Now formatting voting device: /dev/raw/raw2
    Format of 1 voting devices complete.
    Startup will be queued to init within 90 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    CSS is active on these nodes.
    rac1
    CSS is inactive on these nodes.
    rac2
    Local node checking complete.
    Run root.sh on remaining nodes to start CRS daemons.
    [root@rac1 crs]#
    RAC2 root.sh output
    [root@rac2 crs]# ./root.sh
    WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
    WARNING: directory '/u01/crs/oracle/product' is not owned by root
    WARNING: directory '/u01/crs/oracle' is not owned by root
    WARNING: directory '/u01/crs' is not owned by root
    WARNING: directory '/u01' is not owned by root
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up NS directories
    Oracle Cluster Registry configuration upgraded successfully
    WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
    WARNING: directory '/u01/crs/oracle/product' is not owned by root
    WARNING: directory '/u01/crs/oracle' is not owned by root
    WARNING: directory '/u01/crs' is not owned by root
    WARNING: directory '/u01' is not owned by root
    assigning default hostname rac1 for node 1.
    assigning default hostname rac2 for node 2.
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: rac1 rac1-priv rac1
    node 2: rac2 rac2-priv rac2
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Now formatting voting device: /dev/raw/raw2
    Format of 1 voting devices complete.
    Startup will be queued to init within 90 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Failure at final check of Oracle CRS stack.
    10
    [root@rac2 crs]#
    Output of alterrac2.log
    [root@rac2 rac2]# more alertrac2.log
    2009-08-14 23:02:44.699
    [client(5935)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.
    0/crs/log/rac2/client/ocrconfig_5935.log.
    2009-08-14 23:02:44.704
    [client(5935)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.
    0/crs/log/rac2/client/ocrconfig_5935.log.
    2009-08-14 23:02:44.707
    [client(5935)]CRS-1006:The OCR location /dev/raw/raw1 is inaccessible. Details in /u01/crs/oracle/product/10.2.
    0/crs/log/rac2/client/ocrconfig_5935.log.
    2009-08-14 23:02:44.864
    [client(5935)]CRS-1001:The OCR was formatted using version 2.
    2009-08-14 23:02:50.339
    [client(6004)]CRS-1801:Cluster crs configured with nodes rac1 rac2 .
    2009-08-14 23:05:07.603
    [cssd(6600)]CRS-1605:CSSD voting file is online: /dev/raw/raw2. Details in /u01/crs/oracle/product/10.2.0/crs/l
    og/rac2/cssd/ocssd.log.
    [root@rac2 rac2]#
    Since raw devices are not supported from OEL5, I did do the workaround in *63-oracle-raw.rules file under /etc/udev/rules.d* dir.
    ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N"
    ACTION=="add", KERNEL=="sdc1", RUN+="/bin/raw /dev/raw/raw2 %N"
    ACTION=="add", KERNEL=="sdd1", RUN+="/bin/raw /dev/raw/raw3 %N"
    ACTION=="add", KERNEL=="sde1", RUN+="/bin/raw /dev/raw/raw4 %N"
    ACTION=="add", KERNEL=="sdf1", RUN+="/bin/raw /dev/raw/raw5 %N"
    KERNEL=="raw[1-2]*", OWNER="root", GROUP="oinstall", MODE="640"
    KERNEL=="raw[3-5]*", OWNER="oracle", GROUP="oinstall", MODE="640"
    One thing I have noticed after running root.sh on both the nodes is the permissons on raw devices changed from
    Before root.sh
    [root@rac2 crs]# ls -ls /dev/raw*
    0 crw------- 1 root root 162, 0 Aug 14 22:42 /dev/rawctl
    /dev/raw:
    total 0
    0 crw-r----- 1 root oinstall 162, 1 Aug 14 22:42 raw1
    0 crw-r----- 1 root oinstall 162, 2 Aug 14 22:42 raw2
    0 crw-r----- 1 oracle oinstall 162, 3 Aug 14 22:42 raw3
    0 crw-r----- 1 oracle oinstall 162, 4 Aug 14 22:42 raw4
    0 crw-r----- 1 oracle oinstall 162, 5 Aug 14 22:42 raw5
    to
    [root@rac2 crs]# ls -ls /dev/raw*
    0 crw------- 1 root root 162, 0 Aug 14 22:31 /dev/rawctl
    /dev/raw:
    total 0
    0 crw-r----- 1 root oinstall 162, 1 Aug 14 22:56 raw1
    0 crw-r--r-- 1 oracle oinstall 162, 2 Aug 14 23:01 raw2
    0 crw-r----- 1 oracle oinstall 162, 3 Aug 14 22:31 raw3
    0 crw-r----- 1 oracle oinstall 162, 4 Aug 14 22:31 raw4
    0 crw-r----- 1 oracle oinstall 162, 5 Aug 14 22:31 raw5
    [root@rac1 crs]#
    My shared disk listing
    [root@www shared]# ls -ltr
    total 8780
    -rw------- 1 root root 640 Aug 14 22:43 votingdisk.vmdk
    -rw------- 1 root root 598 Aug 14 22:43 ocr.vmdk
    -rw------- 1 root root 604 Aug 14 22:43 asm3.vmdk
    -rw------- 1 root root 604 Aug 14 22:43 asm2.vmdk
    -rw------- 1 root root 604 Aug 14 22:43 asm1.vmdk
    -rw------- 1 root root 65536 Aug 14 22:44 votingdisk-s006.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s005.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s004.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s003.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 votingdisk-s002.vmdk
    -rw------- 1 root root 393216 Aug 14 22:44 votingdisk-s001.vmdk
    -rw------- 1 root root 65536 Aug 14 22:44 ocr-s006.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 ocr-s005.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 ocr-s004.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 ocr-s003.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 ocr-s002.vmdk
    -rw------- 1 root root 393216 Aug 14 22:44 ocr-s001.vmdk
    -rw------- 1 root root 65536 Aug 14 22:44 asm3-s006.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm3-s005.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm3-s004.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm3-s003.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm3-s002.vmdk
    -rw------- 1 root root 393216 Aug 14 22:44 asm3-s001.vmdk
    -rw------- 1 root root 65536 Aug 14 22:44 asm2-s006.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm2-s005.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm2-s004.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm2-s003.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm2-s002.vmdk
    -rw------- 1 root root 393216 Aug 14 22:44 asm2-s001.vmdk
    -rw------- 1 root root 65536 Aug 14 22:44 asm1-s006.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm1-s005.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm1-s004.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm1-s003.vmdk
    -rw------- 1 root root 327680 Aug 14 22:44 asm1-s002.vmdk
    -rw------- 1 root root 393216 Aug 14 22:44 asm1-s001.vmdk
    [root@www shared]#
    I don't know how to fix this problem. I did go through many docs and metalink notes.
    I am new to RAC world. It took 3 days to come to this stage. Please help me.
    Thanks
    Leo

    Hi Surachart,
    Here is my messages output..
    */var/log/messages*
    Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for fe80::20c:29ff:fe6b:f9a8 on eth1.
    Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for 192.168.1.196 on eth1.
    Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for fe80::20c:29ff:fe6b:f99e on eth0.
    Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering new address record for 192.168.0.196 on eth0.
    Aug 20 14:05:01 rac2 avahi-daemon[3627]: Registering HINFO record with values 'I686'/'LINUX'.
    Aug 20 14:05:02 rac2 avahi-daemon[3627]: Server startup complete. Host name is rac2.local. Local service cookie is 927471131.
    Aug 20 14:05:03 rac2 avahi-daemon[3627]: Service "SFTP File Transfer on rac2" (/services/sftp-ssh.service) successfully established.
    Aug 20 14:05:08 rac2 smartd[3739]: smartd version 5.38 [i686-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
    Aug 20 14:05:08 rac2 smartd[3739]: Home page is http://smartmontools.sourceforge.net/
    Aug 20 14:05:08 rac2 smartd[3739]: Opened configuration file /etc/smartd.conf
    Aug 20 14:05:08 rac2 smartd[3739]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/hdc, opened
    Aug 20 14:05:08 rac2 kernel: hdc: drive_cmd: status=0x51 { DriveReady SeekComplete Error }
    Aug 20 14:05:08 rac2 kernel: hdc: drive_cmd: error=0x04 { AbortedCommand }
    Aug 20 14:05:08 rac2 kernel: ide: failed opcode was: 0xec
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/hdc, not ATA, no IDENTIFY DEVICE Structure
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sda, opened
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sda, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on SMART features
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdb, opened
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdb, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdb' to turn on SMART features
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdc, opened
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdc, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdc' to turn on SMART features
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdd, opened
    Aug 20 14:05:08 rac2 smartd[3739]: Device: /dev/sdd, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdd' to turn on SMART features
    Aug 20 14:05:09 rac2 smartd[3739]: Device: /dev/sde, opened
    Aug 20 14:05:09 rac2 smartd[3739]: Device: /dev/sde, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sde' to turn on SMART features
    Aug 20 14:05:10 rac2 smartd[3739]: Device: /dev/sdf, opened
    Aug 20 14:05:10 rac2 smartd[3739]: Device: /dev/sdf, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sdf' to turn on SMART features
    Aug 20 14:05:10 rac2 smartd[3739]: Monitoring 0 ATA and 0 SCSI devices
    Aug 20 14:05:10 rac2 smartd[3741]: smartd has fork()ed into background mode. New PID=3741.
    Aug 20 14:05:13 rac2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
    Aug 20 14:05:13 rac2 last message repeated 3 times
    Aug 20 14:05:27 rac2 gconfd (root-3967): starting (version 2.14.0), pid 3967 user 'root'
    Aug 20 14:05:27 rac2 gconfd (root-3967): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
    Aug 20 14:05:27 rac2 gconfd (root-3967): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 1
    Aug 20 14:05:27 rac2 gconfd (root-3967): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
    Aug 20 14:05:29 rac2 gconfd (root-3967): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 0
    Aug 20 14:05:29 rac2 hald: mounted /dev/hdc on behalf of uid 0
    Aug 20 14:05:29 rac2 hcid[3311]: Default passkey agent (:1.8, /org/bluez/applet) registered
    Aug 20 14:05:31 rac2 nm-system-settings: Loaded plugin ifcfg-rh: (c) 2007 - 2008 Red Hat, Inc. To report bugs please use the NetworkManager mailing list.
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth1 ...
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: read connection 'System eth1'
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-lo ...
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: error: Ignoring loopback device config.
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-eth0 ...
    Aug 20 14:05:31 rac2 nm-system-settings: ifcfg-rh: read connection 'System eth0'
    Aug 20 14:05:31 rac2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
    Aug 20 14:05:32 rac2 last message repeated 4 times
    Aug 20 14:12:51 rac2 kernel: FS-Cache: Loaded
    Aug 20 14:22:06 rac2 xinetd[3488]: START: shell pid=5193 from=192.168.0.195
    Aug 20 14:22:06 rac2 xinetd[3488]: EXIT: shell status=0 pid=5193 duration=0(sec)
    Aug 20 14:22:07 rac2 xinetd[3488]: START: shell pid=5217 from=192.168.0.195
    Aug 20 14:22:07 rac2 xinetd[3488]: EXIT: shell status=0 pid=5217 duration=0(sec)
    Aug 20 14:22:07 rac2 xinetd[3488]: START: shell pid=5241 from=192.168.0.195
    Aug 20 14:22:07 rac2 xinetd[3488]: EXIT: shell status=0 pid=5241 duration=0(sec)
    Aug 20 14:22:16 rac2 xinetd[3488]: EXIT: shell status=0 pid=6236 duration=0(sec)
    Aug 20 14:22:16 rac2 xinetd[3488]: START: shell pid=6265 from=192.168.0.195
    Aug 20 14:22:16 rac2 xinetd[3488]: EXIT: shell status=0 pid=6265 duration=0(sec)
    Aug 20 14:22:16 rac2 xinetd[3488]: START: shell pid=6291 from=192.168.0.195
    Aug 20 14:22:17 rac2 xinetd[3488]: EXIT: shell status=0 pid=6291 duration=1(sec)
    Aug 20 14:22:17 rac2 xinetd[3488]: START: shell pid=6317 from=192.168.0.195
    Aug 20 14:22:17 rac2 xinetd[3488]: EXIT: shell status=0 pid=6317 duration=0(sec)
    [root@rac2 log]#

  • CRS doesn't start

    I added a node to the cluster and end of the installation, crs didn't startup on the added node. Checked the log and has the below. What listener it is looking at? I am on 10gr2.
    2008-04-14 21:50:13.526: [ COMMCRS][2541343872]clsc_connect: (0x641fa0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

    From the working db0 node, here is the o/p. db2 is the new node.
    [oracle@oradbdb0 bin]$ ./crs_stat -t
    Name Type Target State Host
    ora....SM1.asm application ONLINE ONLINE oradbdb0
    ora....0T.lsnr application ONLINE ONLINE oradbdb0
    ora....db0.gsd application ONLINE ONLINE oradbdb0
    ora....db0.ons application ONLINE ONLINE oradbdb0
    ora....db0.vip application ONLINE ONLINE oradbdb0
    ora....SM2.asm application ONLINE ONLINE oradbdb1
    ora....1T.lsnr application ONLINE ONLINE oradbdb1
    ora....db1.gsd application ONLINE ONLINE oradbdb1
    ora....db1.ons application ONLINE OFFLINE
    ora....db1.vip application ONLINE ONLINE oradbdb1
    ora....db2.gsd application ONLINE OFFLINE
    ora....db2.ons application ONLINE OFFLINE
    ora....db2.vip application ONLINE ONLINE oradbdb1

  • RAC, ASM failed to start up on second node , ORA-03113: end-of-file on comm

    i'm installing an RAC with 2 nodes on top of ASM
    when creating ASM Diskgroup , it failed and reported error CRS-0215 failed to start asm on node2
    Oracle 10.2.0.1
    linux CentOs 4.x
    u01/app/oracle/product/10.2.0/db_1/bin/dbca  -progress_only   -configureASM -templateName NO_VALUE -gdbName NO -sid NO      -emConf
    iguration NONE    -diskList /dev/raw/raw2,/dev/raw/raw3  -diskGroupName DATA -datafileJarLocation /u01/app/oracle/product/10.2.0/db_
    1/assistants/dbca/templates  -responseFile NO_VALUE  -nodeinfo node1,node2    -obfuscatedPasswords true   -oratabLocation /u01/app/o
    racle/product/10.2.0/db_1/install/oratab   -asmSysPassword 05dbb0be38ecf8cca822cf3cf99e675448  -redundancy EXTERNA
    [oracle@node2 bin]$ ./crs_stat -t -v
    Name           Type           R/RA   F/FT   Target    State     Host       
    ora....SM1.asm application    0/5    0/0    ONLINE    ONLINE    node1      
    ora....E1.lsnr application    0/5    0/0    ONLINE    ONLINE    node1      
    ora.node1.gsd  application    0/5    0/0    ONLINE    ONLINE    node1      
    ora.node1.ons  application    0/3    0/0    ONLINE    ONLINE    node1      
    ora.node1.vip  application    0/0    0/0    ONLINE    ONLINE    node1      
    ora....SM2.asm application    0/5    0/0    OFFLINE   OFFLINE              
    ora....E2.lsnr application    0/5    0/0    ONLINE    ONLINE    node2      
    ora.node2.gsd  application    0/5    0/0    ONLINE    ONLINE    node2      
    ora.node2.ons  application    0/3    0/0    ONLINE    ONLINE    node2      
    ora.node2.vip  application    0/0    0/0    ONLINE    ONLINE    node2  
    i checked the status , asm is able to start on both nodes if not at the same time ,
    when trying to start the second node , with srvctl or sqlplus , each give the error 03113
    can anyone suggest me of how to bring up both instances ,
    thanks~
    [oracle@node2 bin]$ srvctl stop asm -n node1
    [oracle@node2 bin]$ srvctl start asm -n node1
    [oracle@node2 bin]$ srvctl start asm -n node2
    PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Wed May 27 16:14:50 2009
    node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:Copyright (c) 1982, 2005, Oracle.  All rights reserved.
    node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:Enter user-name: Connected to an idle instance.
    node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:SQL> ORA-03113: end-of-file on communication channel
    node2:ora.node2.ASM2.asm:SQL> Disconnected
    node2:ora.node2.ASM2.asm:
    [code/]
    Edited by: zs_hzh on May 27, 2009 1:25 AM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

    Is it possible to start ASM on second node with SQL*Plus in NOMOUNT state?

  • ORA-27504: IPC error creating OSD context : Unable to start second node

    I have set the DB parameter CLUSTER_INTERCONNECT to point to the Inet addr.
    oifcfg getif
    bondeth0
    172.23.250.128  global  public
    bondib0  192.168.8.0  global
    cluster_interconnect
    When I try to restart the DB services, it is throwing below error while starting the second node.
    These are set of commands I have executed to change the DB Parameter
    alter system set cluster_interconnects =  '192.168.10.6' scope=spfile sid='RAC1' ;
    alter system set cluster_interconnects =  '192.168.10.7' scope=spfile sid='RAC2' ;
    alter system set cluster_interconnects =  '192.168.10.6' scope=spfile sid='ASM1' ;
    alter system set cluster_interconnects =  '192.168.10.7' scope=spfile sid='ASM2' ;
    On second node
    SQL> startup ;
    ORA-27504: IPC error creating OSD context
    ORA-27300: OS system dependent operation:if_not_found failed with status: 0
    ORA-27301: OS failure message: Error 0
    ORA-27302: failure occurred at: skgxpvaddr9
    ORA-27303: additional information: requested interface 192.168.10.6 not found. Check output from ifconfig command
    SQL>
    please let me know whether the proceedure I have followed is wrong
    Thanks

    Node 1:
    [oracle@prdat137db03 etc]$ /sbin/ifconfig bondib0
    bondib0   Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
              inet addr:192.168.10.6  Bcast:192.168.11.255  Mask:255.255.252.0
              inet6 addr: fe80::221:2800:1ef:bc4f/64 Scope:Link
              UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
              RX packets:32550051 errors:0 dropped:0 overruns:0 frame:0
              TX packets:32395961 errors:0 dropped:42 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:19382043590 (18.0 GiB)  TX bytes:17164065360 (15.9 GiB)
    [oracle@prdat137db03 etc]$
    Node 2:
    [oracle@prdat137db04 ~]$ /sbin/ifconfig bondib0
    bondib0   Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
              inet addr:192.168.10.7  Bcast:192.168.11.255  Mask:255.255.252.0
              inet6 addr: fe80::221:2800:1ef:abdb/64 Scope:Link
              UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
              RX packets:29618287 errors:0 dropped:0 overruns:0 frame:0
              TX packets:30769233 errors:0 dropped:12 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:16453595058 (15.3 GiB)  TX bytes:18960175021 (17.6 GiB)
    [oracle@prdat137db04 ~]$

  • 11gr2 crsd core dump during failover or start attempt on second node

    Hi,
    I installed 11gr2 with ASM on one node (solaris SPARC). Then I added another node to this cluster (via addNode.sh script).
    Than got strange error: If my first node is up, second node is started fine and run well. If I shutdown first node - crsd on second node dump to core and fails to restart. I get the same error if I try to start second node when the first one is down.
    In the crsd.log I see the following:
    [  clsdmt][2]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=mskbkp2DBG_CRSD))
    2010-03-03 17:31:35.330: [  clsdmt][2]PID for the Process [18669], connkey 1
    2010-03-03 17:31:35.331: [  clsdmt][2]Creating PID [18669] file for home /u01/grid/11.2.0 host mskbkp2 bin crs to /u01/grid/11
    .2.0/crs/init/
    2010-03-03 17:31:35.331: [  clsdmt][2]Writing PID [18669] to the file [u01/grid/11.2.0/crs/init/mskbkp2.pid]
    2010-03-03 17:31:35.925: [ default][1] CRS Daemon Starting
    2010-03-03 17:31:35.933: [ default][1] ENV Logging level for Module: AGENT 1
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: AGFW 0
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSFRAME 0
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSVER 0
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLUCLS 0
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMCRS 0
    2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMNS 0
    2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSAPP 0
    2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCCL 0
    2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCEVT 0
    2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCOMM 1
    2010-03-03 17:31:35.936: [    CRSD][1] ENV Debug Level(CRSD): 50
    2010-03-03 17:31:35.936: [    CRSD][1] ENV Logging level for Module: CRSD 50
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Debug Level(CRSEVT): 0
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Logging level for Module: CRSEVT 0
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Debug Level(CRSMAIN): 1
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Logging level for Module: CRSMAIN 1
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Debug Level(CRSOCR): 0
    2010-03-03 17:31:35.937: [    CRSD][1] ENV Logging level for Module: CRSOCR 0
    2010-03-03 17:31:35.939: [    CRSD][1] ENV Debug Level(CRSPE): 0
    2010-03-03 17:31:35.939: [    CRSD][1] ENV Logging level for Module: CRSPE 0
    2010-03-03 17:31:35.939: [    CRSD][1] ENV Debug Level(CRSPLACE): 0
    2010-03-03 17:31:35.939: [    CRSD][1] ENV Logging level for Module: CRSPLACE 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Debug Level(CRSRES): 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Logging level for Module: CRSRES 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Debug Level(CRSRPT): 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Logging level for Module: CRSRPT 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Debug Level(CRSRTI): 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Logging level for Module: CRSRTI 0
    2010-03-03 17:31:35.940: [    CRSD][1] ENV Debug Level(CRSSE): 0
    2010-03-03 17:31:35.941: [    CRSD][1] ENV Logging level for Module: CRSSE 0
    2010-03-03 17:31:35.941: [    CRSD][1] ENV Debug Level(CRSSEC): 0
    2010-03-03 17:31:35.941: [    CRSD][1] ENV Logging level for Module: CRSSEC 0
    2010-03-03 17:31:35.941: [    CRSD][1] ENV Debug Level(CRSSHARED): 0
    2010-03-03 17:31:35.941: [    CRSD][1] ENV Logging level for Module: CRSSHARED 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Debug Level(CRSTIMER): 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Logging level for Module: CRSTIMER 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Debug Level(CRSUI): 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Logging level for Module: CRSUI 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Debug Level(CSSCLNT): 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Logging level for Module: CSSCLNT 0
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Debug Level(OCRAPI): 1
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Logging level for Module: OCRAPI 1
    2010-03-03 17:31:35.942: [    CRSD][1] ENV Debug Level(OCRASM): 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Logging level for Module: OCRASM 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Debug Level(OCRCAC): 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Logging level for Module: OCRCAC 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Debug Level(OCRCLI): 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Logging level for Module: OCRCLI 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Debug Level(OCRMAS): 1
    2010-03-03 17:31:35.943: [    CRSD][1] ENV Logging level for Module: OCRMAS 1
    2010-03-03 17:31:35.944: [    CRSD][1] ENV Debug Level(OCRMSG): 1
    2010-03-03 17:31:35.944: [    CRSD][1] ENV Logging level for Module: OCRMSG 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Debug Level(OCROSD): 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Logging level for Module: OCROSD 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Debug Level(OCRRAW): 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Logging level for Module: OCRRAW 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Debug Level(OCRSRV): 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Logging level for Module: OCRSRV 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Debug Level(OCRUTL): 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Logging level for Module: OCRUTL 1
    2010-03-03 17:31:35.945: [    CRSD][1] ENV Debug Level(SuiteTes): 1
    2010-03-03 17:31:35.946: [    CRSD][1] ENV Logging level for Module: SuiteTes 1
    2010-03-03 17:31:35.946: [    CRSD][1] ENV Debug Level(UiServer): 0
    2010-03-03 17:31:35.946: [    CRSD][1] ENV Logging level for Module: UiServer 0
    2010-03-03 17:31:35.946: [ CRSMAIN][1] Checking the OCR device
    2010-03-03 17:31:35.948: [ CRSMAIN][1] Connecting to the CSS Daemon
    2010-03-03 17:31:35.976: [ CRSMAIN][1] Initializing OCR
    2010-03-03 17:31:35.981: [  OCRAPI][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
    2010-03-03 17:31:35.981: [  OCRAPI][1]Check namebufs
    2010-03-03 17:31:35.981: [  OCRAPI][1]Finished checking namebufs
    2010-03-03 17:31:35.982: [    GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
    ], original from [clsss.c : 5026]
    2010-03-03 17:31:36.036: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
    2010-03-03 17:31:36.037: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
    2010-03-03 17:31:36.059: [    GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
    ], original from [clsss.c : 5026]
    2010-03-03 17:31:36.067: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
    fatal if both fail
    2010-03-03 17:31:36.068: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
    f 2: file wallet (LSKP-FSW)
    2010-03-03 17:31:36.068: [    GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
    gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:36.069: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
    2: file wallet (LSKP-FSW) OK
    2010-03-03 17:31:36.069: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
    f 2: OLR wallet (LSKP-CLSW-OLR)
    [   CLWAL][1]clsw_Initialize: OLR initlevel [30000]
    2010-03-03 17:31:36.080: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
    2: OLR wallet (LSKP-CLSW-OLR) OK
    2010-03-03 17:31:36.081: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
    providers - fatal if all fail)
    2010-03-03 17:31:36.081: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:36.152: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:36.152: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:36.152: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:36.172: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
    providers - fatal if all fail)
    2010-03-03 17:31:36.172: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:36.238: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:36.239: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:36.239: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:36.239: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
    2010-03-03 17:31:36.541: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
    (0)
    2010-03-03 17:31:36.552: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
    s (0)
    2010-03-03 17:31:36.771: [  OCRRAW][1]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my v
    otes (0), total votes (0), commit_lsn (9), lsn (9)
    2010-03-03 17:31:36.771: [  OCRRAW][1]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
    2010-03-03 17:31:36.772: [  OCRRAW][1]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
    2010-03-03 17:31:36.772: [  OCRRAW][1]proprioo: 2nd set: (0, 0, 0, 0, 0)
    2010-03-03 17:31:36.830: [  OCRSRV][1]th_init: Successfully retrieved CSS misscount [31].
    2010-03-03 17:31:36.830: [  OCRSRV][1]th_init: Successfully query CLSS mode [3].
    [  OCRMAS][20]th_calc_av:5': Rturn persisted AV [186646784] [11.2.0.1.0]
    2010-03-03 17:31:36.920: [  OCRSRV][20]th_not_master_change: Master change callback not registered
    2010-03-03 17:31:36.920: [  OCRMAS][20]th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number 2
    2010-03-03 17:31:37.134: [  OCRASM][20]proprasmo: ASM cache size is [5MB]
    2010-03-03 17:31:37.142: [  OCRASM][20]proprasmo: ASM cache [5MB] enabled for disk group [DR2_BIN].
    2010-03-03 17:31:37.155: [  OCRRAW][20]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my
    votes (0), total votes (0), commit_lsn (9), lsn (9)
    2010-03-03 17:31:37.155: [  OCRRAW][20]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
    2010-03-03 17:31:37.155: [  OCRRAW][20]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
    2010-03-03 17:31:37.155: [  OCRRAW][20]proprioo: 2nd set: (0, 0, 0, 0, 0)
    2010-03-03 17:31:37.214: [  OCRMAS][20]proath_master:18: Spawned connection mgr thread
    2010-03-03 17:31:37.214: [  OCRMAS][20]proath_master:20: Spawned upgrade thread
    2010-03-03 17:31:37.214: [  OCRMAS][20]th_master:19.1: Wake up upgrade thread
    2010-03-03 17:31:37.216: [  OCRSRV][1]th_snap_local_spawn: Inside snap local spawn. host is [mskbkp2]
    2010-03-03 17:31:37.219: [ CRSMAIN][1] Running as user: root
    2010-03-03 17:31:37.219: [ CRSMAIN][1] CRSD running as the Privileged user
    2010-03-03 17:31:37.219: [  CLSVER][1] Static Version 11.2.0.1.0
    2010-03-03 17:31:37.226: [  OCRMAS][20]th_master:1': Recvd pubdata event from node [2]
    2010-03-03 17:31:37.227: [  OCRMAS][20]th_master:2': Recvd pubdata event for self. Do nothing.
    2010-03-03 17:31:37.227: [  CLSVER][1] Daemon version: 11.2.0.1.0 Software version: 11.2.0.1.0
    2010-03-03 17:31:37.231: [  CLSVER][1] Active Version from OCR:11.2.0.1.0
    2010-03-03 17:31:37.232: [  CLSVER][1] Active Version and Software Version are same
    2010-03-03 17:31:37.232: [  CLSVER][1] Active Version changed to 11.2.0.1.0
    2010-03-03 17:31:37.232: [  OCRSRV][1]th_reg_master_change: Master change callback registered
    2010-03-03 17:31:37.232: [  OCRAPI][1]a_reg_master_change: Registered master change callback
    2010-03-03 17:31:37.232: [  OCRSRV][1]th_not_master_change: Invoking master change callback. Master [2] Inc [1]
    2010-03-03 17:31:37.232: [  OCRAPI][1]a_reg_master_change: Notified master change
    2010-03-03 17:31:37.232: [ CRSMAIN][1] CAA Node Group Pri Data size: 128
    2010-03-03 17:31:37.233: [ CRSMAIN][1] CAA Node Group Pub Data size: 128
    2010-03-03 17:31:37.247: [ CRSMAIN][1] Getting private data of booted nodes
    2010-03-03 17:31:37.247: [ CRSMAIN][1] Checking for booted param on nodenum: 2
    2010-03-03 17:31:37.306: [    CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
    2010-03-03 17:31:37.306: [ CRSMAIN][1] Using Authorizer location: /u01/grid/11.2.0/auth/crs/mskbkp2
    2010-03-03 17:31:37.314: [  OCRSRV][23]th_upgrade: Starting upgrade calculation
    2010-03-03 17:31:37.364: [  CLSCLU][1]clsclu_init: rc 0
    2010-03-03 17:31:37.381: [  OCRSRV][23]th_upgrade:10.1 AV [186646784]. State [11]. Already upgraded.Updated global data to the
    crs version group. Return [0]
    2010-03-03 17:31:37.385: [ CRSMAIN][1] Initializing RTI
    2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing ResouceStateListener
    2010-03-03 17:31:37.433: [CRSTIMER][37] Timer Thread Starting.
    2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing EVMMgr
    2010-03-03 17:31:37.446: [ CRSMAIN][1] Initializing ResourceMap Map
    2010-03-03 17:31:37.461: [ CRSMAIN][1] Subscribing to EVM events for apps
    2010-03-03 17:31:37.504: [ CRSMAIN][1] CRSD locked during state recovery, please wait.
    2010-03-03 17:31:37.516: [ CRSMAIN][1] CRSD recovered, unlocked.
    2010-03-03 17:31:37.525: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
    2010-03-03 17:31:37.525: [ default][1]Check namebufs
    2010-03-03 17:31:37.525: [ default][1]Finished checking namebufs
    2010-03-03 17:31:37.526: [    GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
    ], original from [clsss.c : 5026]
    2010-03-03 17:31:37.569: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
    2010-03-03 17:31:37.569: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
    2010-03-03 17:31:37.587: [    GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
    ], original from [clsss.c : 5026]
    2010-03-03 17:31:37.595: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
    fatal if both fail
    2010-03-03 17:31:37.595: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
    f 2: file wallet (LSKP-FSW)
    2010-03-03 17:31:37.596: [    GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
    gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:37.596: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
    2: file wallet (LSKP-FSW) OK
    2010-03-03 17:31:37.596: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
    f 2: OLR wallet (LSKP-CLSW-OLR)
    [   CLWAL][1]clsw_Initialize: OLR initlevel [30000]
    2010-03-03 17:31:37.607: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
    2: OLR wallet (LSKP-CLSW-OLR) OK
    2010-03-03 17:31:37.607: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
    providers - fatal if all fail)
    2010-03-03 17:31:37.607: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:37.673: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:37.673: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:37.673: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:37.690: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
    providers - fatal if all fail)
    2010-03-03 17:31:37.690: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:37.754: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:37.754: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:37.754: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:37.755: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
    2010-03-03 17:31:37.806: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
    (0)
    2010-03-03 17:31:37.817: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
    s (0)
    2010-03-03 17:31:37.822: [ CRSMAIN][1] CRSD listening on 10 style E2E port (ADDRESS=(PROTOCOL=tcp)(HOST=172.31.25.112)(PORT=38
    983))
    2010-03-03 17:31:37.835: [ CRSMAIN][1] Starting Threads
    2010-03-03 17:31:37.858: [    CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
    2010-03-03 17:31:37.858: [    CRSD][1] AuthLoc /u01/grid/11.2.0/auth/crs/mskbkp2
    2010-03-03 17:31:37.859: [    CRSD][1] PE active version: 11.2.0.1.0
    2010-03-03 17:31:37.859: [    CRSD][1] PE Engine: NEW
    2010-03-03 17:31:37.859: [    CRSD][1] Using OCR batch ops : ENABLED
    2010-03-03 17:31:37.860: [ CRSMAIN][1] Initializing Node Down Monitor
    2010-03-03 17:31:37.860: [ CRSMAIN][1] CRS Daemon Started.
    2010-03-03 17:31:37.860: [    CRSD][1] Connecting to the CSS Daemon
    2010-03-03 17:31:37.861: [    CRSD][1] Local CSS Node Number is: 2
    2010-03-03 17:31:37.863: [    CRSD][1] Local Css Node Name is: mskbkp2
    2010-03-03 17:31:37.863: [    CRSD][1] CRSDPersonality initialized
    2010-03-03 17:31:37.864: [ CRSMAIN][1] Process member data: CRSD:mskbkp2
    2010-03-03 17:31:37.864: [    CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
    2010-03-03 17:31:37.865: [CLSFRAME][1] Inited lsf context 102b3f670
    2010-03-03 17:31:37.865: [CLSFRAME][1] Initing CLS Framework messaging
    2010-03-03 17:31:37.869: [    CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
    2010-03-03 17:31:37.873: [UiServer][1] UI Comms initalize() 1
    2010-03-03 17:31:37.873: [CLSFRAME][1] New Framework state: 2
    2010-03-03 17:31:37.873: [CLSFRAME][1] M2M is starting...
    2010-03-03 17:31:37.873: [  CRSCCL][1]clsCclInit called by process: 18669
    2010-03-03 17:31:37.885: [  CRSCCL][1]USING CLSC ============
    2010-03-03 17:31:37.895: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
    2010-03-03 17:31:37.895: [ default][1]Check namebufs
    2010-03-03 17:31:37.895: [ default][1]Finished checking namebufs
    2010-03-03 17:31:37.950: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
    2010-03-03 17:31:37.951: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
    2010-03-03 17:31:37.970: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
    fatal if both fail
    2010-03-03 17:31:37.970: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
    f 2: file wallet (LSKP-FSW)
    2010-03-03 17:31:37.970: [    GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
    gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:37.970: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
    2: file wallet (LSKP-FSW) OK
    2010-03-03 17:31:37.970: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
    f 2: OLR wallet (LSKP-CLSW-OLR)
    [   CLWAL][1]clsw_Initialize: OLR initlevel [70000]
    2010-03-03 17:31:37.980: [    GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
    2: OLR wallet (LSKP-CLSW-OLR) OK
    2010-03-03 17:31:37.980: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
    providers - fatal if all fail)
    2010-03-03 17:31:37.980: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:38.049: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:38.049: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:38.049: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:38.068: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
    providers - fatal if all fail)
    2010-03-03 17:31:38.068: [    GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
    grid/11.2.0/gpnp/mskbkp2/wallets/peer/
    2010-03-03 17:31:38.134: [    GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
    kbkp2/wallets/peer/cwallet.sso'
    2010-03-03 17:31:38.135: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
    of 2 (LSKP-FSW(1))
    2010-03-03 17:31:38.135: [    GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
    2010-03-03 17:31:38.135: [    GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=3
    2010-03-03 17:31:38.184: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
    (0)
    2010-03-03 17:31:38.194: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
    s (0)
    2010-03-03 17:31:38.200: [  CRSCCL][1]Listening endpoint created sucessfully @ (ADDRESS=(PROTOCOL=tcp)(DEV=54)(HOST=172.31.25.
    112)(PORT=38984)).con = 10359a0d0
    2010-03-03 17:31:38.209: [  CRSCCL][48]CSS Group Registration complete.
    2010-03-03 17:31:38.213: [  CRSCCL][48]cclGetMemberData called
    2010-03-03 17:31:38.215: [  CRSCCL][48]Obtained first membership map.
    2010-03-03 17:31:38.215: [  CRSCCL][48]Dumping member data ------------------
    2010-03-03 17:31:38.215: [  CRSCCL][48]Member (2, 603412550) on node port=.
    2010-03-03 17:31:38.216: [  CRSCCL][48]Done ------------------
    2010-03-03 17:31:38.216: [  CRSCCL][48]Waiting for reconfigs
    2010-03-03 17:31:38.216: [  CRSCCL][49]cclCommunicationHandler started.
    2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: m_pClscCtx=1020c4850m_pUgblm=1035b2a50
    2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: Starting send thread
    2010-03-03 17:31:38.220: [ CRSCOMM][1] IpcL: Listener instantiated for: (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
    2010-03-03 17:31:38.221: [ CRSCOMM][52] Ipc: sendWork thread started.
    2010-03-03 17:31:38.222: [ CRSCOMM][1] IpcL: Listener started listening.
    2010-03-03 17:31:38.223: [ CRSCOMM][53] IpcL: thread started listening
    2010-03-03 17:31:38.223: [CLSFRAME][1] Starting thread model named: AgfwProxySrvTM
    2010-03-03 17:31:38.224: [CLSFRAME][1] Starting thread model named: OcrModuleTM
    2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: PolicyEngineTM
    2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: SharedThreadTM
    2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: UiServerTM
    2010-03-03 17:31:38.225: [CLSFRAME][1] New Framework state: 3
    2010-03-03 17:31:38.227: [  CRSRPT][62] Enabled
    2010-03-03 17:31:38.228: [   CRSPE][61] PE Role|State Update: old role [INVALID] new [INVALID]; old state [Not yet initialized
    ] new [Enabling: waiting for role]
    2010-03-03 17:31:38.229: [   CRSSE][62] Master Change Event; New Master Node ID:2 This Node's ID:2
    2010-03-03 17:31:38.230: [   CRSPE][61] PE Role|State Update: old role [INVALID] new [MASTER]; old state [Enabling: waiting fo
    r role] new [Configuring]
    2010-03-03 17:31:38.230: [   CRSPE][61] PE MASTER NAME: mskbkp2
    2010-03-03 17:31:38.230: [   CRSPE][61] Starting to read configuration
    2010-03-03 17:31:38.260: [   CRSPE][61] Reading (2) servers
    2010-03-03 17:31:38.459: [   CRSPE][61] DM: set global config version to: 150
    2010-03-03 17:31:38.459: [   CRSPE][61] DM: set pool freeze timeout to: 60000
    2010-03-03 17:31:38.459: [   CRSPE][61] DM: Set event seq number to: 13900000
    2010-03-03 17:31:38.459: [   CRSPE][61] DM: Set threshold event seq number to: 13980000
    2010-03-03 17:31:38.460: [   CRSPE][61] Sent request to write event sequence number 14000000 to repository
    2010-03-03 17:31:38.483: [   CRSPE][61] Wrote new event sequence to repository
    2010-03-03 17:31:38.568: [   CRSPE][61] Reading (15) types
    2010-03-03 17:31:38.593: [   CRSPE][61] Reading (3) server pools
    2010-03-03 17:31:38.624: [   CRSPE][61] Reading (21) resources
    2010-03-03 17:31:39.987: [   CRSPE][61] Finished reading configuration. Parsing...
    2010-03-03 17:31:39.988: [   CRSPE][61] Parsing resource types...
    2010-03-03 17:31:40.030: [    CRSD][61] Initializing the config version for type ora.asm.type to: 1
    2010-03-03 17:31:40.035: [    CRSD][61] Initializing the config version for type ora.cluster_resource.type to: 1
    2010-03-03 17:31:40.040: [    CRSD][61] Initializing the config version for type ora.cluster_vip.type to: 1
    2010-03-03 17:31:40.044: [    CRSD][61] Initializing the config version for type ora.cluster_vip_net1.type to: 1
    2010-03-03 17:31:40.048: [    CRSD][61] Dump State Starting ...
    2010-03-03 17:31:40.048: [    CRSD][61] State Dump for RTILock
    2010-03-03 17:31:40.048: [    CRSD][61] Lock State List is busy, skipping ..
    2010-03-03 17:31:40.048: [    CRSD][61] State Dump for Timer
    2010-03-03 17:31:40.049: [    CRSD][61] Timer map size=0
    2010-03-03 17:31:40.049: [   CRSPE][61] Dumping PE Data Model...:DM has [0 resources][0 types][0 servers][0 spools]
    ------------- RESOURCES:
    ------------- TYPES:
    ------------- SERVERS:
    ------------- SERVER POOLS:
    2010-03-03 17:31:40.049: [   CRSPE][61] Dumping ICE contents...:ICE operation count: 0
    2010-03-03 17:31:40.049: [    CRSD][61] Dump State Done.
    I guess that there is some thing wrong in configuration, but cannot find out what.
    Any help would be appreciated.
    Thanks

    Hi,
    Please check your disk attributes and permission of OCR/Voting and other ASM devices. The disk attribute should be changed to be shared among all nodes of cluster. It happened with us in 10.2.0.4 where disk was not shared and we were able to start crs from only one node at a time so please check disk attributes. Please see blog keyurmakwanacrs.blogspot.com for AIX which we faced. Not surle whether you've similar problem or not. We had 10.2.0.4 clusterware.
    thanks,
    Keyur

  • Public Interface not responding after second node is started in the cluster

    Hi
    Has anyone ever experienced the public interface not responding between nodes in the cluster (ping, ssh, scp) after the second nodeapps is started in the cluster?
    This is a new install so all I have installed so far is the base release of CRS 10.2.0. This is on Solaris10. The vipca failed during the installation, however I was able to proceed and manually add the nodeapps using srvctl add nodeaps -n -o -A.
    It seems after the second node is started I loose all connectivity to the public interfaces and to my default gateway.
    Also I'm getting the following messages sometimes after I try and stop the nodeapps and start them back up.
    CRS-1006: No more members to consider
    CRS-0215: Could not start resource 'ora.node1.vip'.
    Any suggestions on where I should start troubleshooting?
    Thanks

    Do you have default GW?
    It can connects with GW, can't it?
    Check metalink
    CRS-0215: Could not start resource 'ora..vip' [ID 356535.1]
    CRS-1006: No more members to consider when starting service [ID 465364.1]
    Good Luck

  • Root.sh failed on second node while installing CRS 10g on centos 5.5

    root.sh failed on second node while installing CRS 10g
    Hi all,
    I am able to install Oracle 10g RAC clusterware on first node of the cluster. However, when I run the root.sh script as root
    user on second node of the cluster, it fails with following error message:
    NO KEYS WERE WRITTEN. Supply -force parameter to override.
    -force is destructive and will destroy any previous cluster
    configuration.
    Oracle Cluster Registry for cluster has already been initialized
    Startup will be queued to init within 90 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Failure at final check of Oracle CRS stack.
    10
    and run cluvfy stage -post hwos -n all -verbose,it show message:
    ERROR:
    Could not find a suitable set of interfaces for VIPs.
    Result: Node connectivity check failed.
    Checking shared storage accessibility...
    Disk Sharing Nodes (2 in count)
    /dev/sda db2 db1
    and run cluvfy stage -pre crsinst -n all -verbose,it show message:
    ERROR:
    Could not find a suitable set of interfaces for VIPs.
    Result: Node connectivity check failed.
    Checking system requirements for 'crs'...
    No checks registered for this product.
    and run cluvfy stage -post crsinst -n all -verbose,it show message:
    Result: Node reachability check passed from node "DB2".
    Result: User equivalence check passed for user "oracle".
    Node Name CRS daemon CSS daemon EVM daemon
    db2 no no no
    db1 yes yes yes
    Check: Health of CRS
    Node Name CRS OK?
    db1 unknown
    Result: CRS health check failed.
    check crsd.log and show message:
    clsc_connect: (0x143ca610) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_db2_crs))
    clsssInitNative: connect failed, rc 9
    Any help would be greatly appreciated.
    Edited by: 868121 on 2011-6-24 上午12:31

    Hello, it took a little searching, but I found this in a note in the GRID installation guide for Linux/UNIX:
    Public IP addresses and virtual IP addresses must be in the same subnet.
    In your case, you are using two different subnets for the VIPs.

  • My ipod nano 6th gen doesn't start. When I plug it, only blink the apple logo on the screen. I try to reset it holding by 8 seconds start button and volume down and doesn't work. My Ipod doesn't have any shock or water damage. How can I do?

    My ipod nano 6th gen doesn't start. When I plug it, only blink the apple logo on the screen. I try to reset it holding by 8 seconds start button and volume down and doesn't work. My Ipod doesn't have any shock or water damage. How can I do?

    Hey rulosrojos,
    The article linked below details a number of troubleshooting steps that can help restore functionality to your iPod nano.
    iPod nano (6th generation): Hardware troubleshooting
    https://support.apple.com/en-us/TS3474
    Sincerely,
    Allen

  • When i double-click the mozilla icon, it doesn't start. In the task manager i noted that the process "firefox.exe" appears and desappear after 5 seconds. The problem remains in mozilla safe-mode, but not in windows safe-mode. Thks!

    Yesterday i installed the firefox 6.0 on my windows 7 computer. When i double-click the mozilla icon, it doesn't start. I opened the task manager and noted that the process "firefox.exe" appears and desappear after 5 seconds. I tried to start mozilla in safe mode but the problem remains. The only way to use mozilla in my computer is restarting the windows in safe-mode. Last thing: i already tried to reinstall the program. Please help me!

    I'm also using Windows 7 on a Lenovo U310 laptop.

  • How to start CRS and other services after rebooting nodes.

    Hi,
    i have created two node cluster database.how to start CRS and other service after rebooting nodes?
    Thanks,

    use crsctl command start...
    $ORA_CRS_HOME/bin/crsctl start crsOr check status about CLusterware...
    $ORA_CRS_HOME/bin/crsctl check crs
    CSS appears healthy
    CRS appears healthy
    EVM appears healthyIf not OK... check log at $ORA_CRS_HOME/log/hostname/*
    and check status database , service and etc...
    use crs_stat
    $ORA_CRS_HOME/bin/crs_stat -t
    $ORA_CRS_HOME/bin/crs_stat if you find some service or instance ... not online...
    use srvctl command, help to start
    Get Help...
    $ORA_CRS_HOME/bin/srvctl -h

  • Can't start instance on the second node

    I install RAC on Redhat AS 2.1 with LVM.
    After run "srvctl start db -d RAC" or "srvctl start instance -d RAC -i RAC2",
    the instance on node2 is not running.
    The error message say that the instance is already running even I just start RAC2 only. But the RAC1 runs well.
    I don't know it is the file system or the environment problem.
    Thanks

    I installed Oracle 9ir2 RAC on two RH advance server 2.1. There are some error on creating database, OLAP and ODM.
    Starting db instance on node1 is fine, but the second node is not running. Then I install database on the second node
    With software only(to get sqlplus to work), and start instance by SQL. I found the SGA is not the same.
    To get my SCSI and LVM to work, I recompiler the kernel. Is that possible two different kernel image to make SGA has
    Different size? Will the SGA size be the same on Real Application Cluster? I am sure the sem, shmmax have the same
    Value on both nodes. How can I change the role of node1 to node2 manually in ocm?
    Thanks

  • Clusterware doesn't start. Nothing written to logfile.

    Hi,
    2 node rac on 11g r1 Linux
    Clusterware on one node starts up without issue.
    But on the second node, it doesn't come up. None of the Logfiles are updated. (no timestamp change at all)
    #./crsctl start crs
    Same status.
    How to debug, if nothing is written to Log file
    thx
    sunil

    Check <GRID_HOME>log/<node_name>/alert<nodename>.log.. that should give your further information... if you dont see anything in alert log, find most recent file in GRID_HOME/log/nodename directory.

Maybe you are looking for

  • Cannot find file .dat

    After using perl script, will produce three files 1. file .ctl 2. file .log 3. file .dis When using sqlloader to load file , error messages founds " file .dat cannot be found ". I see that the data that to loaded to table is on file .dis. I usually e

  • Printing after editing

    I'm still very new to "iPhoto 5" and can not figure out or find in "iPhoto" "Help" how to print or e-mail a picture that I have edited! Should I make a dupicate of the edited picture and then print that? When I click on the "dupicate" option where wi

  • Selection conditions in Infopackage

    Hi,   I developed a data source in R/3 used to load Text Data. I selected the Key field,Key text and language fields from the table and i checked them in the extract structure of data source. Now when i replicated the data source and created info pac

  • I have a itunes card that isn't activated and I don't have the receipt.

    I got an Itunes card a while back and I am just now using it, it says it's not activated. I tried other questions and they said to go back to the store with the receipt, the thing it, I don't have a receipt.

  • Intelligent IP monitoring

    hey everyone i'm going to create a Intelligent IP monitoring with netbeans and i dont know how to start is there anyone can give me some source i can refer to? or if anyone can help me out here that would be great. its a router/Switch monitoring syst