Second node cannot bring up clusterware stack
Hi forum,
I have a two-node RAC in VirtualBox VMs. After booting the first node (regardless which of the two), the complete Clusterware stack is started successfully. When booting the other node, the Clusterware stack does not come up successfully. When I try to start components manually, there seems to be a circular dependency between CRS/CSSD and ASM.
Configuration:
OS: OEL 5.8, UEK
Grid: 11.2
Database: 11.2.0.1
Any help is appreciated!
Thanks,
Andreas
Hi,
What is the error you have in the OHASD.log and CRSD.log. There are dependencies of the component from 11gR2.
http://levipereira.wordpress.com/2012/01/11/explaining-how-to-store-ocr-voting-disks-and-asm-spfile-on-asm-diskgroup-rac-or-rac-extended/
Anand
Similar Messages
-
0x800713CF: The owner node cannot run this resource (SQL)
I have SQL 2012 SP2 service on 2 FC nodes of 2012 R2 servers. Everything were working fine about a year, but now my first Node got mixed up with updates and it is offline now. Problem is, that second Node cannot start SQL services By its own.
I checked/tried:
- I checked, that other services related to SQL are online.
- I chenged Node´s ownership to SQL service
- Cluster name has full Access rights to OU where all Computer record of this service are in place
The error I get:
0x800713cf:The cluster resource cannot be brought online. The owner node cannot run this resource.Thank you both. I´m not sure, how to to check the workload. In FC graphical GUI, both nodes are memebers in owner group, same Place where the owner order can be changed or owner can be disabled. I gone through entire FC management snap-in
and didn´t find any other places or settings to go through. The article kb313882 Umesh refers, is for Server 2008, and I have 2012 R2. Command
cluster res <var> name of resource</var> /listowners is not supported (there is no such command). I got first node up and running, and then SQL services were ON again, but if I shutdown first Node, SQL services will go down
and won´t come up. Disk storage are online and SP level matches.
Please could you guide me step By step, how to check ownership in 2012 R2? -
11gR2 clusterware installation problem on root.sh script on second node
Hi all,
I wanna install the *11gR2 RAC* on ORA-Linux 5.5 (x86_64) using VMware server but on the second node i get two "*failed*" at the end of root.sh script.
After that i try to install DB but ı can see only one node.What is the problem...
I will send the output, ı need your help.
Thank you all for helping..
Hosts file:(we have no ping problem )
[root@rac2 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
# Public
192.168.2.101 rac1.localdomain rac1
192.168.2.102 rac2.localdomain rac2
# Private
192.168.0.101 rac1-priv.localdomain rac1-priv
192.168.0.102 rac2-priv.localdomain rac2-priv
# Virtual
192.168.2.111 rac1-vip.localdomain rac1-vip
192.168.2.112 rac2-vip.localdomain rac2-vip
# SCAN
192.168.2.201 rac-scan.localdomain rac-scan
[root@rac2 ~]#
FIRST NODE root.sh script output...
[root@rac2 ~]# /u01/app/11.2.0/db_1/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-12-06 14:45:06: Parsing the host name
2010-12-06 14:45:06: Checking for super user privileges
2010-12-06 14:45:06: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/db_1/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
ASM created and started successfully.
DiskGroup DATA created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 587cc69413ce4fd3bf0c2c2548fb9017.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
1. ONLINE 587cc69413ce4fd3bf0c2c2548fb9017 (/dev/oracleasm/disks/DISK1) [DATA]
Located 1 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
CRS-2677: Stop of 'ora.crsd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2'
CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac2'
CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2'
CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac2'
CRS-2676: Start of 'ora.DATA.dg' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac2'
CRS-2676: Start of 'ora.registry.acfs' on 'rac2' succeeded
rac2 2010/12/06 14:52:06 /u01/app/11.2.0/db_1/cdata/rac2/backup_20101206_145206.olr
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 6847 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[root@rac2 ~]#
SECOND NODE root.sh script output
[root@rac1 db_1]# ./root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-12-06 14:54:11: Parsing the host name
2010-12-06 14:54:11: Checking for super user privileges
2010-12-06 14:54:11: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/db_1/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
ASM created and started successfully.
DiskGroup DATA created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
Successful addition of voting disk 2761ce8d47b44fbabf73462151e3ba1d.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
1. ONLINE 2761ce8d47b44fbabf73462151e3ba1d (/dev/oracleasm/disks/DISK1) [DATA]
Located 1 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'
CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded
PRCR-1079 : *Failed* to start resource ora.scan1.vip
CRS-5005: IP Address: 192.168.2.201 is already in use in the network
CRS-2674: Start of 'ora.scan1.vip' on 'rac1' *failed*
CRS-2632: There are no more servers to try to place resource 'ora.scan1.vip' on that would satisfy its placement policy
start scan ... *failed*
Configure Oracle Grid Infrastructure for a Cluster ... *failed*
Updating inventory properties for clusterware
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 6847 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[root@rac1 db_1]# * "./runcluvfy.sh stage -pre -crsinst -n rac1,rac2 " outputs are same each node....*
[oracle@rac2 grid]$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2
Performing pre-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "rac2"
Checking user equivalence...
User equivalence check passed for user "oracle"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Node connectivity passed for subnet "192.168.2.0" with node(s) rac2,rac1
TCP connectivity check passed for subnet "192.168.2.0"
Node connectivity passed for subnet "192.168.122.0" with node(s) rac2,rac1
TCP connectivity check failed for subnet "192.168.122.0"
Node connectivity passed for subnet "192.168.0.0" with node(s) rac2,rac1
TCP connectivity check passed for subnet "192.168.0.0"
Interfaces found on subnet "192.168.2.0" that are likely candidates for VIP are:
rac2 eth0:192.168.2.102 eth0:192.168.2.112 eth0:192.168.2.201
rac1 eth0:192.168.2.101 eth0:192.168.2.111
Interfaces found on subnet "192.168.122.0" that are likely candidates for a private interconnect are:
rac2 virbr0:192.168.122.1
rac1 virbr0:192.168.122.1
Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect are:
rac2 eth1:192.168.0.102
rac1 eth1:192.168.0.101
Node connectivity check passed
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "rac2:/tmp"
Free disk space check passed for "rac1:/tmp"
User existence check passed for "oracle"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "oracle" in group "oinstall" [as Primary] passed
Membership check for user "oracle" in group "dba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "make-3.81"
Package existence check passed for "binutils-2.17.50.0.6"
Package existence check passed for "gcc-4.1.2"
Package existence check passed for "libaio-0.3.106 (i386)"
Package existence check passed for "libaio-0.3.106 (x86_64)"
Package existence check passed for "glibc-2.5-24 (i686)"
Package existence check passed for "glibc-2.5-24 (x86_64)"
Package existence check passed for "compat-libstdc++-33-3.2.3 (i386)"
Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)"
Package existence check passed for "elfutils-libelf-0.125 (x86_64)"
Package existence check passed for "elfutils-libelf-devel-0.125"
Package existence check passed for "glibc-common-2.5"
Package existence check passed for "glibc-devel-2.5 (i386)"
Package existence check passed for "glibc-devel-2.5 (x86_64)"
Package existence check passed for "glibc-headers-2.5"
Package existence check passed for "gcc-c++-4.1.2"
Package existence check passed for "libaio-devel-0.3.106 (i386)"
Package existence check passed for "libaio-devel-0.3.106 (x86_64)"
Package existence check passed for "libgcc-4.1.2 (i386)"
Package existence check passed for "libgcc-4.1.2 (x86_64)"
Package existence check passed for "libstdc++-4.1.2 (i386)"
Package existence check passed for "libstdc++-4.1.2 (x86_64)"
Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)"
Package existence check passed for "sysstat-7.0.2"
Package existence check passed for "unixODBC-2.2.11 (i386)"
Package existence check passed for "unixODBC-2.2.11 (x86_64)"
Package existence check passed for "unixODBC-devel-2.2.11 (i386)"
Package existence check passed for "unixODBC-devel-2.2.11 (x86_64)"
Package existence check passed for "ksh-20060214"
Check for multiple users with UID value 0 passed
Current group ID check passed
Core file name pattern consistency check passed.
User "oracle" is not part of "root" group. Check passed
Default user file creation mask check passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
NTP Configuration file check passed
Checking daemon liveness...
Liveness check passed for "ntpd"
NTP daemon slewing option check passed
NTP daemon's boot time configuration check for slewing option passed
NTP common Time Server Check started...
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Clock time offset check passed
Clock synchronization check using Network Time Protocol(NTP) passed
Pre-check for cluster services setup was successful.
[oracle@rac2 grid]$ I'm confused :)
Edited by: Eren GULERYUZ on 06.Ara.2010 05:57Hi,
it looks like, that your "shared device" you are using is not really shared.
The second node does "create an ASM diskgroup" and create OCR and Voting disks. If this indeed would be a shared device, he should have recognized, that your disk is shared.
So as a result your VMware configuration must be wrong, and the disk you presented as shared disk is not really shared.
Which VMWare version did you use? It will not work correctly with the workstation or player edition, since shared disks are only really working with the server version.
If you indeed using the server, could you paste your vm configurations?
Furthermore I recommend using Virtual Box. There is a nice how-to:
http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnOEL5UsingVirtualBox.php
Sebastian -
Timed out waiting for the CRS stack to start on Second node.
Hi,
I am trying to setup 2 node 11gR2 RAC in vmware. I face the Timed out waiting for the CRS stack to start error on second node while running root.sh.I checked the cluster log files located in /u01/app/11.2.0/grid/log/node1/alertnode2.log it shows as mentioned below.But when i logged into ASM instance and checked the diskgroup and it is in mount state on both nodes,So i am really confused why i ended up with error in 2nd node on running root.sh. Can anyone tell me how to correct this error or any other checking needs to be done?
/u01/app/11.2.0/grid/log/node1/alertnode2.log
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:10.314
[crsd(25309)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
2014-01-21 21:40:11.217
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:12.422
[crsd(25335)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
2014-01-21 21:40:13.327
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:14.514
[crsd(25356)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
2014-01-21 21:40:15.439
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:15.440
[ohasd(23261)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
root.sh output
[root@node2 ~]# /u01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2014-01-21 21:37:55: Parsing the host name
2014-01-21 21:37:55: Checking for super user privileges
2014-01-21 21:37:55: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_p
arams
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node 11gdb, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'node2'
CRS-2676: Start of 'ora.mdnsd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'node2'
CRS-2676: Start of 'ora.gipcd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node2'
CRS-2676: Start of 'ora.gpnpd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node2'
CRS-2676: Start of 'ora.cssdmonitor' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node2'
CRS-2672: Attempting to start 'ora.diskmon' on 'node2'
CRS-2676: Start of 'ora.diskmon' on 'node2' succeeded
CRS-2676: Start of 'ora.cssd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'node2'
CRS-2676: Start of 'ora.ctssd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'node2'
CRS-2676: Start of 'ora.drivers.acfs' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node2'
CRS-2676: Start of 'ora.asm' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'node2'
CRS-2676: Start of 'ora.crsd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'node2'
CRS-2676: Start of 'ora.evmd' on 'node2' succeeded
Timed out waiting for the CRS stack to start.
Regards,
007Hi,
Are you trying to install on Vmware Workstation or Vmware Server?
If you are using vmware server add below line in your .vmx file for both node
scsi1.sharedBus = "VIRTUAL"
First check which scsi serial number are you using as above I have selected scsi1 serial during disk addition, -
The Script root.sh problem - ora.asm and ASM and Clusterware Stack failed
Folks,
Hello. I am installing Oracle 11gR2 RAC using 2 VMs (rac1 and rac2) whose OS are Oracle Linux 5.6 in VMPlayer according to the website http://appsdbaworkshop.blogspot.com/2011/10/11gr2-rac-on-linux-56-using-vmware.html
I am installing Grid infrastructure. On step 9 of 10 - execute script /u01/app/grid/root.sh for 2 VMs rac1 and rac2.
After run root.sh in rac1 successfully. I run root.sh in rac2 and get an error as below:
[root@rac2 grid]# ./root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= ora11g
ORACLE_HOME= /u01/app/grid
Enter the full pathname of the local bin directory: [usr/local/bin]: /usr/local/bin
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2012-03-05 16:32:52: Parsing the host name
2012-03-05 16:32:52: Checking for super user privileges
2012-03-05 16:32:52: User has super user privileges
Using configuration parameter file: /u01/app/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
Start action for octssd aborted
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac2'
CRS-2672: Attempting to start 'ora.asm' on 'rac2'
CRS-2676: Start of 'ora.drivers.acfs' on 'rac2' succeeded
CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
CRS-2664: Resource 'ora.ctssd' is already running on 'rac2'
CRS-4000: Command Start failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/grid/bin/crsctl start resource ora.asm -init
Start of resource "ora.asm -init" failed
Failed to start ASM
Failed to start Oracle Clusterware stack
[root@rac2 grid]#
As we see the output above, at the end of the output
1) Start of resource ora.asm -init failed
2) Failed to start ASM
3) Failed to start Oracle Clusterware stack
The runInstaller is in the first VM rac1. My question is:
Do any folk understand how to solve the script root.sh in rac2 problem ( 3 fails of ora.asm, ASM and Clusterware stack as above) ?
Thanks.Please check there is no firewall exist:
try this like:
root.sh fails on second node
MOS note:
11gR2 Grid: root.sh Fails to Start the Clusterware on the Second Node Due to Firewall on Private Network [ID 981357.1]
Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1] (Most probabily this issue) -
Error after running root.sh on second node
hi,
i have installed clusterware on a 2 node system running on RHEL 5
i followed the prereqs , and solved all the errors i encountered
after clusterware installation , it asks to run root.sh on all the nodes
when i ran root.sh on second node
it gave this error
Running vipca(silent) for configuring nodeapps
/home/oracle/crs/oracle/product/10/crs/jdk/jre//bin/java: error while loading
shared libraries: libpthread.so.0: cannot open shared object file:
No such file or directory
so i follwed metalink note 414163.1
after that i called it a day
in the morning , wheni started both nodes
and started vipca on second node
it gave this error
PRKH:1010 unable to communicate with crs services
the i ran ps-ef | grep crs
root 3201 1 0 15:37 ? 00:00:00 /bin/sh /etc/init.d/init.crsd run
crsctl check crs gave
failure 1 contacting css daemon
cannot communicate with crs
cannot communicate with evm
what should i do to start these services?crsd and cssd logs were empty and there was no relevant info in crs alert
i am just reinstalling clusterware now
one thing i wanted to ask
why does owner ship of raw files change back to root (after node restart)
even though i chnged them to oracle -
Error CLSRSC-507 during the execution of root.sh on second node
Hi all.
OS.......: Red-Hat 6.5
RDBMS: Oracle 12.1.0.2.0
During the installation of a 2-node RAC in a RHEL 6.5, during the execution of the root.sh script in the second node, i get the following error:
[root@oraprd02 grid]# ./root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2015/05/04 22:47:16 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2015/05/04 22:47:59 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2015/05/04 22:48:00 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
2015/05/04 22:48:46 CLSRSC-507: The root script cannot proceed on this node oraprd02 because either the first-node operations have not completed on node oraprd01 or there was an error in obtaining the status of the first-node operations.
Died at /u01/app/12.1.0/grid/crs/install/crsutils.pm line 3681.
The command '/u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/crs/install /u01/app/12.1.0/grid/crs/install/rootcrs.pl ' execution failed
The root.sh on the first node completed successfully. I get the succeeded message from the script in the first node.
Have anyone faced this problem? Any assistance will be most helpfull.
Thanks in advance.crsd and cssd logs were empty and there was no relevant info in crs alert
i am just reinstalling clusterware now
one thing i wanted to ask
why does owner ship of raw files change back to root (after node restart)
even though i chnged them to oracle -
Root.sh failed on second node while installing CRS 10g on centos 5.5
root.sh failed on second node while installing CRS 10g
Hi all,
I am able to install Oracle 10g RAC clusterware on first node of the cluster. However, when I run the root.sh script as root
user on second node of the cluster, it fails with following error message:
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Failure at final check of Oracle CRS stack.
10
and run cluvfy stage -post hwos -n all -verbose,it show message:
ERROR:
Could not find a suitable set of interfaces for VIPs.
Result: Node connectivity check failed.
Checking shared storage accessibility...
Disk Sharing Nodes (2 in count)
/dev/sda db2 db1
and run cluvfy stage -pre crsinst -n all -verbose,it show message:
ERROR:
Could not find a suitable set of interfaces for VIPs.
Result: Node connectivity check failed.
Checking system requirements for 'crs'...
No checks registered for this product.
and run cluvfy stage -post crsinst -n all -verbose,it show message:
Result: Node reachability check passed from node "DB2".
Result: User equivalence check passed for user "oracle".
Node Name CRS daemon CSS daemon EVM daemon
db2 no no no
db1 yes yes yes
Check: Health of CRS
Node Name CRS OK?
db1 unknown
Result: CRS health check failed.
check crsd.log and show message:
clsc_connect: (0x143ca610) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_db2_crs))
clsssInitNative: connect failed, rc 9
Any help would be greatly appreciated.
Edited by: 868121 on 2011-6-24 上午12:31Hello, it took a little searching, but I found this in a note in the GRID installation guide for Linux/UNIX:
Public IP addresses and virtual IP addresses must be in the same subnet.
In your case, you are using two different subnets for the VIPs. -
Root.sh failed at second node OUL 6.3 Oracle GRID 11.2.0.3
Hi, im installing a two node cluster mounted on Oracle Linux 6.3 with Oracle DB 11.2.0.3, the installation went smooth up until the execution of the root.sh script on the second node.
THe script return this final lines:
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node nodo1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Start of resource "ora.crsd" failed
CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'nodo2'
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Grid Infrastructure stack
Failed to start Cluster Ready Services at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1286.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
In $GRID_HOME/log/node2/alertnode.log It appears to be a Cluster Time Synchronization Service issue, (i didn't synchronyze the nodes..) however the CTSS is running in observer mode, wich i believe it shouldn't affect the installation process. After that i lost it...there's an entry CRS-5018 indicating that an unused HAIP route was removed... and then, out of the blue: CRS-5818:Aborted command 'start' for resource 'ora.asm'. Some clarification will be deeply apreciated.
Here's the complete log:
2013-04-01 13:39:35.358
[client(12163)]CRS-2101:The OLR was formatted using version 3.
2013-04-01 19:40:19.597
[ohasd(12338)]CRS-2112:The OLR service started on node nodo2.
2013-04-01 19:40:19.657
[ohasd(12338)]CRS-1301:Oracle High Availability Service started on node nodo2.
[client(12526)]CRS-10001:01-Apr-13 13:41 ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.39-400.17.2.el6uek.i686'
[client(12528)]CRS-10001:01-Apr-13 13:41 ACFS-9201: Not Supported
[client(12603)]CRS-10001:01-Apr-13 13:41 ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.39-400.17.2.el6uek.i686'
2013-04-01 19:41:17.509
[ohasd(12338)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2013-04-01 19:41:17.618
[gpnpd(12695)]CRS-2328:GPNPD started on node nodo2.
2013-04-01 19:41:21.363
[cssd(12755)]CRS-1713:CSSD daemon is started in exclusive mode
2013-04-01 19:41:23.194
[ohasd(12338)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2013-04-01 19:41:56.144
[cssd(12755)]CRS-1707:Lease acquisition for node nodo2 number 2 completed
2013-04-01 19:41:57.545
[cssd(12755)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/ASM_DISK_1; details in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log.
[cssd(12755)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node nodo1 and is terminating; details at (:CSSNM00006:) in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log
2013-04-01 19:41:58.549
[ohasd(12338)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'nodo2'.
2013-04-01 19:42:10.025
[gpnpd(12695)]CRS-2329:GPNPD on node nodo2 shutdown.
2013-04-01 19:42:11.407
[mdnsd(12685)]CRS-5602:mDNS service stopping by request.
2013-04-01 19:42:29.642
[gpnpd(12947)]CRS-2328:GPNPD started on node nodo2.
2013-04-01 19:42:33.241
[cssd(13012)]CRS-1713:CSSD daemon is started in clustered mode
2013-04-01 19:42:35.104
[ohasd(12338)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2013-04-01 19:42:44.065
[cssd(13012)]CRS-1707:Lease acquisition for node nodo2 number 2 completed
2013-04-01 19:42:45.484
[cssd(13012)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/ASM_DISK_1; details in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log.
2013-04-01 19:42:52.138
[cssd(13012)]CRS-1601:CSSD Reconfiguration complete. Active nodes are nodo1 nodo2 .
2013-04-01 19:42:55.081
[ctssd(13076)]CRS-2403:The Cluster Time Synchronization Service on host nodo2 is in observer mode.
2013-04-01 19:42:55.581
[ctssd(13076)]CRS-2401:The Cluster Time Synchronization Service started on host nodo2.
2013-04-01 19:42:55.581
[ctssd(13076)]CRS-2407:The new Cluster Time Synchronization Service reference node is host nodo1.
2013-04-01 19:43:08.875
[ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
2013-04-01 19:43:08.876
[ctssd(13076)]CRS-2409:The clock on host nodo2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2013-04-01 19:43:13.565
[u01/app/11.2.0/grid/bin/orarootagent.bin(13064)]CRS-5018:(:CLSN00037:) Removed unused HAIP route: 169.254.0.0 / 255.255.0.0 / 0.0.0.0 / eth0
2013-04-01 19:53:09.800
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5818:Aborted command 'start' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:0:223} in /u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log.
2013-04-01 19:53:11.827
[ohasd(12338)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) {0:0:223} in /u01/app/11.2.0/grid/log/nodo2/ohasd/ohasd.log.
2013-04-01 19:53:12.779
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:53:13.892
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:53:43.877
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:54:13.891
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:54:43.906
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:55:13.914
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:55:43.918
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:56:13.922
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:56:53.209
[crsd(13741)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:07:01.128
[crsd(13741)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:07:01.278
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:07:08.689
[crsd(15248)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:13:10.138
[ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
2013-04-01 20:17:13.024
[crsd(15248)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:17:13.171
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:17:20.826
[crsd(16746)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:27:25.020
[crsd(16746)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:27:25.176
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:27:31.591
[crsd(18266)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:37:35.668
[crsd(18266)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:37:35.808
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:37:43.209
[crsd(19762)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:43:11.160
[ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
2013-04-01 20:47:47.487
[crsd(19762)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:47:47.637
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:47:55.086
[crsd(21242)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:57:59.343
[crsd(21242)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:57:59.492
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:58:06.996
[crsd(22744)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:08:11.046
[crsd(22744)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:08:11.192
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:08:18.726
[crsd(24260)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:13:12.000
[ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
2013-04-01 21:18:22.262
[crsd(24260)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:18:22.411
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:18:29.927
[crsd(25759)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:28:34.467
[crsd(25759)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:28:34.616
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:28:41.990
[crsd(27291)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:38:45.012
[crsd(27291)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:38:45.160
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:38:52.790
[crsd(28784)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:43:12.378
[ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
2013-04-01 21:48:56.285
[crsd(28784)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:48:56.435
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:49:04.421
[crsd(30272)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:59:08.183
[crsd(30272)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:59:08.318
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:59:15.860
[crsd(31772)]CRS-1012:The OCR service started on node nodo2.Hi santysharma, thanks for the reply, i have two ethernet interfaces: eth0 (public network 192.168.1.0) and eth1 (private network 10.5.3.0), there is no device using that ip range, here's the output of route command:
(Sorry for the alignment, i tried to tab it but the editor trims it again)
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
private * 255.255.255.0 U 0 0 0 eth1
link-local * 255.255.0.0 U 1002 0 0 eth0
link-local * 255.255.0.0 U 1003 0 0 eth1
public * 255.255.255.0 U 0 0 0 eth0
And the /etc/hosts file
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.5.3.1 nodo1.cluster nodo1
10.5.3.2 nodo2.cluster nodo2
192.168.1.13 cluster-scan
192.168.1.14 nodo1-vip
192.168.1.15 nodo2-vip
And the ifconfig -a
eth0 Link encap:Ethernet HWaddr C8:3A:35:D9:C6:2B
inet addr:192.168.1.12 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::ca3a:35ff:fed9:c62b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:34708 errors:0 dropped:18 overruns:0 frame:0
TX packets:24693 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:48545969 (46.2 MiB) TX bytes:1994381 (1.9 MiB)
eth1 Link encap:Ethernet HWaddr 00:0D:87:D0:A3:8E
inet addr:10.5.3.2 Bcast:10.5.3.255 Mask:255.255.255.0
inet6 addr: fe80::20d:87ff:fed0:a38e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:5344 (5.2 KiB)
Interrupt:23 Base address:0x6000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:20 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1320 (1.2 KiB) TX bytes:1320 (1.2 KiB)
Now that i'm thinking i've read somewhere that ipv6 was no supported...yet there's no relation with the 169.254.x.x ip range. -
Root.sh fails on second node
I already posted this issue on database installation forum, and was suggested to post it on this forum.
Here are the details.
I am running Linux 64bit on ESx clients. Installing Oracle 11gR2.
It passed all the per-requisite. Run root.sh on first node. It finished with no errorrs.
On second node I got the following:
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-07-13 12:51:28: Parsing the host name
2010-07-13 12:51:28: Checking for super user privileges
2010-07-13 12:51:28: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node fred0224, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'fred0225'
CRS-2676: Start of 'ora.mdnsd' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'fred0225'
CRS-2676: Start of 'ora.gipcd' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'fred0225'
CRS-2676: Start of 'ora.gpnpd' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'fred0225'
CRS-2676: Start of 'ora.cssdmonitor' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'fred0225'
CRS-2672: Attempting to start 'ora.diskmon' on 'fred0225'
CRS-2676: Start of 'ora.diskmon' on 'fred0225' succeeded
CRS-2676: Start of 'ora.cssd' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'fred0225'
Start action for octssd aborted
CRS-2676: Start of 'ora.ctssd' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'fred0225'
CRS-2672: Attempting to start 'ora.asm' on 'fred0225'
CRS-2676: Start of 'ora.drivers.acfs' on 'fred0225' succeeded
CRS-2676: Start of 'ora.asm' on 'fred0225' succeeded
CRS-2664: Resource 'ora.ctssd' is already running on 'fred0225'
CRS-4000: Command Start failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.asm -init
Start of resource "ora.asm -init" failed
Failed to start ASM
Failed to start Oracle Clusterware stack
In the ocssd.log I found
[ CSSD][3559689984]clssnmvDHBValidateNCopy: node 1, fred0224, has a disk HB, but no network HB, DHB has rcfg 174483948, wrtcnt, 232, LATS 521702664, lastSeqNo 232, uniqueness 1279039649, timestamp 1279039959/521874274
In oraagent_oracle.log I found
[ clsdmc][1212365120]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD)) with status 9
2010-07-13 12:54:07.234: [ora.gpnpd][1212365120] [check] Error = error 9 encountered when connecting to GPNPD
2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Calling PID check for daemon
2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Trying to check PID = 20584
2010-07-13 12:54:07.432: [ COMMCRS][1285794112]clsc_connect: (0x1304d850) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD))
[ clsdmc][1222854976]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD)) with status 9
2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Error = error 9 encountered when connecting to MDNSD
2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Calling PID check for daemon
2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Trying to check PID = 20571
2010-07-13 12:54:08.841: [ COMMCRS][1201875264]clsc_connect: (0x12f3b1d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD))
[ clsdmc][1159915840]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD)) with status 9
2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Error = error 9 encountered when connecting to GIPCD
2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Calling PID check for daemon
2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Trying to check PID = 20566
2010-07-13 12:54:10.242: [ COMMCRS][1254324544]clsc_connect: (0x12f35630) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD))
In oracssdagent_root.log I found
2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clssscConnect: gipc request failed with 29 (0x16)
2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clsssInitNative: connect failed, rc 29
2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clssnsqlnum: RPC failed rc 3
2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_cssini: failed 3 to fetch node number
2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_init: css init done, nodenum -1.
2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssRecvMsg: got a disconnect from the server while waiting for message type 43
2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssGetNLSData: Failure receiving a msg, rc 3
If you need more info, let me know.Well, the error clearly indicates that a communication problem exists on the private interconnect.
Could this be a setting in ESX, which prevents some communication between the clients on the second network card? Any routing table in ESX not configured correctly?
Sebastian -
Root.sh on second node fails
I am running Linux 64bit. Installing Oracle 11gR2.
It passed all the per-requisite. Run root.sh on first node. It finished with no errorrs.
On second node I got the following:
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-07-13 12:51:28: Parsing the host name
2010-07-13 12:51:28: Checking for super user privileges
2010-07-13 12:51:28: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node fred0224, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'fred0225'
CRS-2676: Start of 'ora.mdnsd' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'fred0225'
CRS-2676: Start of 'ora.gipcd' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'fred0225'
CRS-2676: Start of 'ora.gpnpd' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'fred0225'
CRS-2676: Start of 'ora.cssdmonitor' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'fred0225'
CRS-2672: Attempting to start 'ora.diskmon' on 'fred0225'
CRS-2676: Start of 'ora.diskmon' on 'fred0225' succeeded
CRS-2676: Start of 'ora.cssd' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'fred0225'
Start action for octssd aborted
CRS-2676: Start of 'ora.ctssd' on 'fred0225' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'fred0225'
CRS-2672: Attempting to start 'ora.asm' on 'fred0225'
CRS-2676: Start of 'ora.drivers.acfs' on 'fred0225' succeeded
CRS-2676: Start of 'ora.asm' on 'fred0225' succeeded
CRS-2664: Resource 'ora.ctssd' is already running on 'fred0225'
CRS-4000: Command Start failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.asm -init
Start of resource "ora.asm -init" failed
Failed to start ASM
Failed to start Oracle Clusterware stack
In the ocssd.log I found
[ CSSD][3559689984]clssnmvDHBValidateNCopy: node 1, fred0224, has a disk HB, but no network HB, DHB has rcfg 174483948, wrtcnt, 232, LATS 521702664, lastSeqNo 232, uniqueness 1279039649, timestamp 1279039959/521874274
In oraagent_oracle.log I found
[ clsdmc][1212365120]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD)) with status 9
2010-07-13 12:54:07.234: [ora.gpnpd][1212365120] [check] Error = error 9 encountered when connecting to GPNPD
2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Calling PID check for daemon
2010-07-13 12:54:07.238: [ora.gpnpd][1212365120] [check] Trying to check PID = 20584
2010-07-13 12:54:07.432: [ COMMCRS][1285794112]clsc_connect: (0x1304d850) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GPNPD))
[ clsdmc][1222854976]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD)) with status 9
2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Error = error 9 encountered when connecting to MDNSD
2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Calling PID check for daemon
2010-07-13 12:54:08.649: [ora.mdnsd][1222854976] [check] Trying to check PID = 20571
2010-07-13 12:54:08.841: [ COMMCRS][1201875264]clsc_connect: (0x12f3b1d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_MDNSD))
[ clsdmc][1159915840]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD)) with status 9
2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Error = error 9 encountered when connecting to GIPCD
2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Calling PID check for daemon
2010-07-13 12:54:10.051: [ora.gipcd][1159915840] [check] Trying to check PID = 20566
2010-07-13 12:54:10.242: [ COMMCRS][1254324544]clsc_connect: (0x12f35630) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=fred0225DBG_GIPCD))
In oracssdagent_root.log I found
2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clssscConnect: gipc request failed with 29 (0x16)
2010-07-13 12:52:28.698: [ CSSCLNT][1102481728]clsssInitNative: connect failed, rc 29
2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clssnsqlnum: RPC failed rc 3
2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_cssini: failed 3 to fetch node number
2010-07-13 12:53:55.222: [ USRTHRD][1102481728] clsnomon_init: css init done, nodenum -1.
2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssRecvMsg: got a disconnect from the server while waiting for message type 43
2010-07-13 12:53:55.222: [ CSSCLNT][1102481728]clsssGetNLSData: Failure receiving a msg, rc 3
If anyone needs more info please let me know.On all nodes,
1. Modify the /etc/sysconfig/oracleasm with:
ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"
2. restart the asmlib by :
# /etc/init.d/oracleasm restart
3. Run root.sh on the 2nd node
hope this helps you -
11gr2 crsd core dump during failover or start attempt on second node
Hi,
I installed 11gr2 with ASM on one node (solaris SPARC). Then I added another node to this cluster (via addNode.sh script).
Than got strange error: If my first node is up, second node is started fine and run well. If I shutdown first node - crsd on second node dump to core and fails to restart. I get the same error if I try to start second node when the first one is down.
In the crsd.log I see the following:
[ clsdmt][2]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=mskbkp2DBG_CRSD))
2010-03-03 17:31:35.330: [ clsdmt][2]PID for the Process [18669], connkey 1
2010-03-03 17:31:35.331: [ clsdmt][2]Creating PID [18669] file for home /u01/grid/11.2.0 host mskbkp2 bin crs to /u01/grid/11
.2.0/crs/init/
2010-03-03 17:31:35.331: [ clsdmt][2]Writing PID [18669] to the file [u01/grid/11.2.0/crs/init/mskbkp2.pid]
2010-03-03 17:31:35.925: [ default][1] CRS Daemon Starting
2010-03-03 17:31:35.933: [ default][1] ENV Logging level for Module: AGENT 1
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: AGFW 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSFRAME 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSVER 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLUCLS 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMCRS 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMNS 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSAPP 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCCL 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCEVT 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCOMM 1
2010-03-03 17:31:35.936: [ CRSD][1] ENV Debug Level(CRSD): 50
2010-03-03 17:31:35.936: [ CRSD][1] ENV Logging level for Module: CRSD 50
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSEVT): 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSEVT 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSMAIN): 1
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSMAIN 1
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSOCR): 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSOCR 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Debug Level(CRSPE): 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Logging level for Module: CRSPE 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Debug Level(CRSPLACE): 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Logging level for Module: CRSPLACE 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRES): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRES 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRPT): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRPT 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRTI): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRTI 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSSE): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSE 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Debug Level(CRSSEC): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSEC 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Debug Level(CRSSHARED): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSHARED 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CRSTIMER): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CRSTIMER 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CRSUI): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CRSUI 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CSSCLNT): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CSSCLNT 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(OCRAPI): 1
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: OCRAPI 1
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(OCRASM): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRASM 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRCAC): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRCAC 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRCLI): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRCLI 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRMAS): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRMAS 1
2010-03-03 17:31:35.944: [ CRSD][1] ENV Debug Level(OCRMSG): 1
2010-03-03 17:31:35.944: [ CRSD][1] ENV Logging level for Module: OCRMSG 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCROSD): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCROSD 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRRAW): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRRAW 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRSRV): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRSRV 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRUTL): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRUTL 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(SuiteTes): 1
2010-03-03 17:31:35.946: [ CRSD][1] ENV Logging level for Module: SuiteTes 1
2010-03-03 17:31:35.946: [ CRSD][1] ENV Debug Level(UiServer): 0
2010-03-03 17:31:35.946: [ CRSD][1] ENV Logging level for Module: UiServer 0
2010-03-03 17:31:35.946: [ CRSMAIN][1] Checking the OCR device
2010-03-03 17:31:35.948: [ CRSMAIN][1] Connecting to the CSS Daemon
2010-03-03 17:31:35.976: [ CRSMAIN][1] Initializing OCR
2010-03-03 17:31:35.981: [ OCRAPI][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:35.981: [ OCRAPI][1]Check namebufs
2010-03-03 17:31:35.981: [ OCRAPI][1]Finished checking namebufs
2010-03-03 17:31:35.982: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
], original from [clsss.c : 5026]
2010-03-03 17:31:36.036: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:36.037: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:36.059: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
], original from [clsss.c : 5026]
2010-03-03 17:31:36.067: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:36.068: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:36.068: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.069: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:36.069: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [30000]
2010-03-03 17:31:36.080: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:36.081: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:36.081: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:36.172: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:36.172: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.238: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
2010-03-03 17:31:36.541: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:36.552: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:36.771: [ OCRRAW][1]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my v
otes (0), total votes (0), commit_lsn (9), lsn (9)
2010-03-03 17:31:36.771: [ OCRRAW][1]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:36.772: [ OCRRAW][1]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:36.772: [ OCRRAW][1]proprioo: 2nd set: (0, 0, 0, 0, 0)
2010-03-03 17:31:36.830: [ OCRSRV][1]th_init: Successfully retrieved CSS misscount [31].
2010-03-03 17:31:36.830: [ OCRSRV][1]th_init: Successfully query CLSS mode [3].
[ OCRMAS][20]th_calc_av:5': Rturn persisted AV [186646784] [11.2.0.1.0]
2010-03-03 17:31:36.920: [ OCRSRV][20]th_not_master_change: Master change callback not registered
2010-03-03 17:31:36.920: [ OCRMAS][20]th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number 2
2010-03-03 17:31:37.134: [ OCRASM][20]proprasmo: ASM cache size is [5MB]
2010-03-03 17:31:37.142: [ OCRASM][20]proprasmo: ASM cache [5MB] enabled for disk group [DR2_BIN].
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my
votes (0), total votes (0), commit_lsn (9), lsn (9)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: 2nd set: (0, 0, 0, 0, 0)
2010-03-03 17:31:37.214: [ OCRMAS][20]proath_master:18: Spawned connection mgr thread
2010-03-03 17:31:37.214: [ OCRMAS][20]proath_master:20: Spawned upgrade thread
2010-03-03 17:31:37.214: [ OCRMAS][20]th_master:19.1: Wake up upgrade thread
2010-03-03 17:31:37.216: [ OCRSRV][1]th_snap_local_spawn: Inside snap local spawn. host is [mskbkp2]
2010-03-03 17:31:37.219: [ CRSMAIN][1] Running as user: root
2010-03-03 17:31:37.219: [ CRSMAIN][1] CRSD running as the Privileged user
2010-03-03 17:31:37.219: [ CLSVER][1] Static Version 11.2.0.1.0
2010-03-03 17:31:37.226: [ OCRMAS][20]th_master:1': Recvd pubdata event from node [2]
2010-03-03 17:31:37.227: [ OCRMAS][20]th_master:2': Recvd pubdata event for self. Do nothing.
2010-03-03 17:31:37.227: [ CLSVER][1] Daemon version: 11.2.0.1.0 Software version: 11.2.0.1.0
2010-03-03 17:31:37.231: [ CLSVER][1] Active Version from OCR:11.2.0.1.0
2010-03-03 17:31:37.232: [ CLSVER][1] Active Version and Software Version are same
2010-03-03 17:31:37.232: [ CLSVER][1] Active Version changed to 11.2.0.1.0
2010-03-03 17:31:37.232: [ OCRSRV][1]th_reg_master_change: Master change callback registered
2010-03-03 17:31:37.232: [ OCRAPI][1]a_reg_master_change: Registered master change callback
2010-03-03 17:31:37.232: [ OCRSRV][1]th_not_master_change: Invoking master change callback. Master [2] Inc [1]
2010-03-03 17:31:37.232: [ OCRAPI][1]a_reg_master_change: Notified master change
2010-03-03 17:31:37.232: [ CRSMAIN][1] CAA Node Group Pri Data size: 128
2010-03-03 17:31:37.233: [ CRSMAIN][1] CAA Node Group Pub Data size: 128
2010-03-03 17:31:37.247: [ CRSMAIN][1] Getting private data of booted nodes
2010-03-03 17:31:37.247: [ CRSMAIN][1] Checking for booted param on nodenum: 2
2010-03-03 17:31:37.306: [ CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.306: [ CRSMAIN][1] Using Authorizer location: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.314: [ OCRSRV][23]th_upgrade: Starting upgrade calculation
2010-03-03 17:31:37.364: [ CLSCLU][1]clsclu_init: rc 0
2010-03-03 17:31:37.381: [ OCRSRV][23]th_upgrade:10.1 AV [186646784]. State [11]. Already upgraded.Updated global data to the
crs version group. Return [0]
2010-03-03 17:31:37.385: [ CRSMAIN][1] Initializing RTI
2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing ResouceStateListener
2010-03-03 17:31:37.433: [CRSTIMER][37] Timer Thread Starting.
2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing EVMMgr
2010-03-03 17:31:37.446: [ CRSMAIN][1] Initializing ResourceMap Map
2010-03-03 17:31:37.461: [ CRSMAIN][1] Subscribing to EVM events for apps
2010-03-03 17:31:37.504: [ CRSMAIN][1] CRSD locked during state recovery, please wait.
2010-03-03 17:31:37.516: [ CRSMAIN][1] CRSD recovered, unlocked.
2010-03-03 17:31:37.525: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:37.525: [ default][1]Check namebufs
2010-03-03 17:31:37.525: [ default][1]Finished checking namebufs
2010-03-03 17:31:37.526: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
], original from [clsss.c : 5026]
2010-03-03 17:31:37.569: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:37.569: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:37.587: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
], original from [clsss.c : 5026]
2010-03-03 17:31:37.595: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:37.595: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [30000]
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:37.690: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:37.690: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:37.755: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
2010-03-03 17:31:37.806: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:37.817: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:37.822: [ CRSMAIN][1] CRSD listening on 10 style E2E port (ADDRESS=(PROTOCOL=tcp)(HOST=172.31.25.112)(PORT=38
983))
2010-03-03 17:31:37.835: [ CRSMAIN][1] Starting Threads
2010-03-03 17:31:37.858: [ CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.858: [ CRSD][1] AuthLoc /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.859: [ CRSD][1] PE active version: 11.2.0.1.0
2010-03-03 17:31:37.859: [ CRSD][1] PE Engine: NEW
2010-03-03 17:31:37.859: [ CRSD][1] Using OCR batch ops : ENABLED
2010-03-03 17:31:37.860: [ CRSMAIN][1] Initializing Node Down Monitor
2010-03-03 17:31:37.860: [ CRSMAIN][1] CRS Daemon Started.
2010-03-03 17:31:37.860: [ CRSD][1] Connecting to the CSS Daemon
2010-03-03 17:31:37.861: [ CRSD][1] Local CSS Node Number is: 2
2010-03-03 17:31:37.863: [ CRSD][1] Local Css Node Name is: mskbkp2
2010-03-03 17:31:37.863: [ CRSD][1] CRSDPersonality initialized
2010-03-03 17:31:37.864: [ CRSMAIN][1] Process member data: CRSD:mskbkp2
2010-03-03 17:31:37.864: [ CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:37.865: [CLSFRAME][1] Inited lsf context 102b3f670
2010-03-03 17:31:37.865: [CLSFRAME][1] Initing CLS Framework messaging
2010-03-03 17:31:37.869: [ CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:37.873: [UiServer][1] UI Comms initalize() 1
2010-03-03 17:31:37.873: [CLSFRAME][1] New Framework state: 2
2010-03-03 17:31:37.873: [CLSFRAME][1] M2M is starting...
2010-03-03 17:31:37.873: [ CRSCCL][1]clsCclInit called by process: 18669
2010-03-03 17:31:37.885: [ CRSCCL][1]USING CLSC ============
2010-03-03 17:31:37.895: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:37.895: [ default][1]Check namebufs
2010-03-03 17:31:37.895: [ default][1]Finished checking namebufs
2010-03-03 17:31:37.950: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:37.951: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:38.068: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:38.068: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:38.134: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=3
2010-03-03 17:31:38.184: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:38.194: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:38.200: [ CRSCCL][1]Listening endpoint created sucessfully @ (ADDRESS=(PROTOCOL=tcp)(DEV=54)(HOST=172.31.25.
112)(PORT=38984)).con = 10359a0d0
2010-03-03 17:31:38.209: [ CRSCCL][48]CSS Group Registration complete.
2010-03-03 17:31:38.213: [ CRSCCL][48]cclGetMemberData called
2010-03-03 17:31:38.215: [ CRSCCL][48]Obtained first membership map.
2010-03-03 17:31:38.215: [ CRSCCL][48]Dumping member data ------------------
2010-03-03 17:31:38.215: [ CRSCCL][48]Member (2, 603412550) on node port=.
2010-03-03 17:31:38.216: [ CRSCCL][48]Done ------------------
2010-03-03 17:31:38.216: [ CRSCCL][48]Waiting for reconfigs
2010-03-03 17:31:38.216: [ CRSCCL][49]cclCommunicationHandler started.
2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: m_pClscCtx=1020c4850m_pUgblm=1035b2a50
2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: Starting send thread
2010-03-03 17:31:38.220: [ CRSCOMM][1] IpcL: Listener instantiated for: (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:38.221: [ CRSCOMM][52] Ipc: sendWork thread started.
2010-03-03 17:31:38.222: [ CRSCOMM][1] IpcL: Listener started listening.
2010-03-03 17:31:38.223: [ CRSCOMM][53] IpcL: thread started listening
2010-03-03 17:31:38.223: [CLSFRAME][1] Starting thread model named: AgfwProxySrvTM
2010-03-03 17:31:38.224: [CLSFRAME][1] Starting thread model named: OcrModuleTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: PolicyEngineTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: SharedThreadTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: UiServerTM
2010-03-03 17:31:38.225: [CLSFRAME][1] New Framework state: 3
2010-03-03 17:31:38.227: [ CRSRPT][62] Enabled
2010-03-03 17:31:38.228: [ CRSPE][61] PE Role|State Update: old role [INVALID] new [INVALID]; old state [Not yet initialized
] new [Enabling: waiting for role]
2010-03-03 17:31:38.229: [ CRSSE][62] Master Change Event; New Master Node ID:2 This Node's ID:2
2010-03-03 17:31:38.230: [ CRSPE][61] PE Role|State Update: old role [INVALID] new [MASTER]; old state [Enabling: waiting fo
r role] new [Configuring]
2010-03-03 17:31:38.230: [ CRSPE][61] PE MASTER NAME: mskbkp2
2010-03-03 17:31:38.230: [ CRSPE][61] Starting to read configuration
2010-03-03 17:31:38.260: [ CRSPE][61] Reading (2) servers
2010-03-03 17:31:38.459: [ CRSPE][61] DM: set global config version to: 150
2010-03-03 17:31:38.459: [ CRSPE][61] DM: set pool freeze timeout to: 60000
2010-03-03 17:31:38.459: [ CRSPE][61] DM: Set event seq number to: 13900000
2010-03-03 17:31:38.459: [ CRSPE][61] DM: Set threshold event seq number to: 13980000
2010-03-03 17:31:38.460: [ CRSPE][61] Sent request to write event sequence number 14000000 to repository
2010-03-03 17:31:38.483: [ CRSPE][61] Wrote new event sequence to repository
2010-03-03 17:31:38.568: [ CRSPE][61] Reading (15) types
2010-03-03 17:31:38.593: [ CRSPE][61] Reading (3) server pools
2010-03-03 17:31:38.624: [ CRSPE][61] Reading (21) resources
2010-03-03 17:31:39.987: [ CRSPE][61] Finished reading configuration. Parsing...
2010-03-03 17:31:39.988: [ CRSPE][61] Parsing resource types...
2010-03-03 17:31:40.030: [ CRSD][61] Initializing the config version for type ora.asm.type to: 1
2010-03-03 17:31:40.035: [ CRSD][61] Initializing the config version for type ora.cluster_resource.type to: 1
2010-03-03 17:31:40.040: [ CRSD][61] Initializing the config version for type ora.cluster_vip.type to: 1
2010-03-03 17:31:40.044: [ CRSD][61] Initializing the config version for type ora.cluster_vip_net1.type to: 1
2010-03-03 17:31:40.048: [ CRSD][61] Dump State Starting ...
2010-03-03 17:31:40.048: [ CRSD][61] State Dump for RTILock
2010-03-03 17:31:40.048: [ CRSD][61] Lock State List is busy, skipping ..
2010-03-03 17:31:40.048: [ CRSD][61] State Dump for Timer
2010-03-03 17:31:40.049: [ CRSD][61] Timer map size=0
2010-03-03 17:31:40.049: [ CRSPE][61] Dumping PE Data Model...:DM has [0 resources][0 types][0 servers][0 spools]
------------- RESOURCES:
------------- TYPES:
------------- SERVERS:
------------- SERVER POOLS:
2010-03-03 17:31:40.049: [ CRSPE][61] Dumping ICE contents...:ICE operation count: 0
2010-03-03 17:31:40.049: [ CRSD][61] Dump State Done.
I guess that there is some thing wrong in configuration, but cannot find out what.
Any help would be appreciated.
ThanksHi,
Please check your disk attributes and permission of OCR/Voting and other ASM devices. The disk attribute should be changed to be shared among all nodes of cluster. It happened with us in 10.2.0.4 where disk was not shared and we were able to start crs from only one node at a time so please check disk attributes. Please see blog keyurmakwanacrs.blogspot.com for AIX which we faced. Not surle whether you've similar problem or not. We had 10.2.0.4 clusterware.
thanks,
Keyur -
Why root.sh fails in the second node?
Hi
After successful install of oracle11 grid in 2 nodes and running root.sh on
node1, root.sh on node2 fails:
[root@vmorarac2 dev]#
/u01/app/product/11.2.0/oracle/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/product/11.2.0/oracle
Enter the full pathname of the local bin directory:
[/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed
by
Database Configuration Assistant when a database is
created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2013-07-17 08:37:10: Parsing the host name
2013-07-17 08:37:10: Checking for super user
privileges
2013-07-17 08:37:10: User has super user privileges
Using configuration parameter file:
/u01/app/product/11.2.0/oracle/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been
started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but
found an active CSS daemon on node vmorarac1, number 1, and is terminating
CRS-2673: Attempting to stop 'ora.cssdmonitor' on
'vmorarac2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'vmorarac2'
succeeded
An active cluster was found during exclusive startup,
restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on
'vmorarac2'
CRS-2676: Start of 'ora.mdnsd' on 'vmorarac2'
succeeded
CRS-2672: Attempting to start 'ora.gipcd' on
'vmorarac2'
CRS-2676: Start of 'ora.gipcd' on 'vmorarac2'
succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on
'vmorarac2'
CRS-2676: Start of 'ora.gpnpd' on 'vmorarac2'
succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on
'vmorarac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'vmorarac2'
succeeded
CRS-2672: Attempting to start 'ora.cssd' on
'vmorarac2'
CRS-2672: Attempting to start 'ora.diskmon' on
'vmorarac2'
CRS-2676: Start of 'ora.diskmon' on 'vmorarac2'
succeeded
CRS-2674: Start of 'ora.cssd' on 'vmorarac2' failed
CRS-2679: Attempting to clean 'ora.cssd' on
'vmorarac2'
CRS-2681: Clean of 'ora.cssd' on 'vmorarac2' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on
'vmorarac2'
CRS-2677: Stop of 'ora.diskmon' on 'vmorarac2'
succeeded
CRS-4000: Command Start failed, or completed with
errors.
CRS-2672: Attempting to start 'ora.cssd' on
'vmorarac2'
CRS-2672: Attempting to start 'ora.diskmon' on
'vmorarac2'
CRS-2674: Start of 'ora.diskmon' on 'vmorarac2' failed
CRS-2679: Attempting to clean 'ora.diskmon' on
'vmorarac2'
CRS-5016: Process
"/u01/app/product/11.2.0/oracle/bin/diskmon" spawned by agent
"/u01/app/product/11.2.0/oracle/bin/orarootagent.bin" for action "clean" failed:
details at "(:CLSN00010:)" in
"/u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/orarootagent_root.log"
CRS-2681: Clean of 'ora.diskmon' on 'vmorarac2'
succeeded
CRS-2674: Start of 'ora.cssd' on 'vmorarac2' failed
CRS-2679: Attempting to clean 'ora.cssd' on
'vmorarac2'
CRS-2681: Clean of 'ora.cssd' on 'vmorarac2' succeeded
CRS-4000: Command Start failed, or completed with
errors.
Command return code of 1 (256) from command:
/u01/app/product/11.2.0/oracle/bin/crsctl start resource ora.ctssd -init -env
USR_ORA_ENV=CTSS_REBOOT=TRUE
Start of resource "ora.ctssd -init -env
USR_ORA_ENV=CTSS_REBOOT=TRUE" failed
Failed to start CTSS
Failed to start Oracle Clusterware stack
[root@vmorarac2 dev]#
[root@vmorarac2 dev]#
try again:
[root@vmorarac2 bin]# ./crsctl start resource ora.ctssd
-init -env USR_ORA_ENV=CTSS_REBOOT=TRUE
CRS-2672: Attempting to start 'ora.cssd' on
'vmorarac2'
CRS-2672: Attempting to start 'ora.diskmon' on
'vmorarac2'
CRS-2674: Start of 'ora.diskmon' on 'vmorarac2' failed
CRS-2679: Attempting to clean 'ora.diskmon' on
'vmorarac2'
CRS-5016: Process
"/u01/app/product/11.2.0/oracle/bin/diskmon" spawned by agent
"/u01/app/product/11.2.0/oracle/bin/orarootagent.bin" for action "clean" failed:
details at "(:CLSN00010:)" in
"/u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/orarootagent_root.log"
CRS-2681: Clean of 'ora.diskmon' on 'vmorarac2'
succeeded
CRS-2674: Start of 'ora.cssd' on 'vmorarac2' failed
CRS-2679: Attempting to clean 'ora.cssd' on
'vmorarac2'
CRS-2681: Clean of 'ora.cssd' on 'vmorarac2' succeeded
CRS-4000: Command Start failed, or completed with
errors.
[root@vmorarac2 bin]# ps -ef | grep u01
root 8913 1 0 08:37 ? 00:00:06
/u01/app/product/11.2.0/oracle/bin/ohasd.bin reboot
oracle 10507 1 0 08:39 ? 00:00:02
/u01/app/product/11.2.0/oracle/bin/oraagent.bin
oracle 10522 1 0 08:39 ? 00:00:00
/u01/app/product/11.2.0/oracle/bin/mdnsd.bin
oracle 10534 1 0 08:39 ? 00:00:00
/u01/app/product/11.2.0/oracle/bin/gipcd.bin
oracle 10548 1 0 08:39 ? 00:00:39
/u01/app/product/11.2.0/oracle/bin/gpnpd.bin
root 11723 1 0 11:00 ? 00:00:03
/u01/app/product/11.2.0/oracle/bin/cssdmonitor
[oracle@vmorarac2 bin]$ ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready
Services
CRS-4530: Communications failure contacting Cluster
Synchronization Services daemon
CRS-4534: Cannot communicate with Event Managerlog:
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2013-07-17 08:38:04.485: [ AGFW][3481860944] Starting the agent: /u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/
2013-07-17 08:38:04.485: [ AGENT][3481860944] Agent framework initialized, Process Id = 10319
2013-07-17 08:38:04.487: [ USRTHRD][3481860944] Utils::getCrsHome crsHome /u01/app/product/11.2.0/oracle
2013-07-17 08:38:04.487: [ USRTHRD][3481860944] Process::convertPidToString pid = 10319
2013-07-17 08:38:04.488: [ AGFW][3481860944] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
2013-07-17 08:38:04.488: [CLSFRAME][3481860944] Inited lsf context 0x317e9e0
2013-07-17 08:38:04.488: [CLSFRAME][3481860944] Initing CLS Framework messaging
2013-07-17 08:38:04.488: [CLSFRAME][3481860944] New Framework state: 2
2013-07-17 08:38:04.488: [CLSFRAME][3481860944] M2M is starting...
2013-07-17 08:38:04.490: [ CRSCOMM][3481860944] m_pClscCtx=0x31d1bd0m_pUgblm=0x31d5720
2013-07-17 08:38:04.490: [ CRSCOMM][3481860944] Starting send thread
2013-07-17 08:38:04.490: [ CRSCOMM][1119435072] clsIpc: sendWork thread started.
2013-07-17 08:38:04.491: [ CRSCOMM][1129924928] IPC Client thread started listening
2013-07-17 08:38:04.491: [ CRSCOMM][1129924928] init data sent from server
2013-07-17 08:38:04.491: [CLSFRAME][3481860944] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:vmorarac2
2013-07-17 08:38:04.491: [CLSFRAME][3481860944] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:vmorarac2
2013-07-17 08:38:04.492: [CLSFRAME][3481860944] Starting thread model named: MultiThread
2013-07-17 08:38:04.492: [CLSFRAME][3481860944] Starting thread model named: SingleThread
2013-07-17 08:38:04.492: [CLSFRAME][3481860944] Starting thread model named: SingleThreadT
2013-07-17 08:38:04.492: [CLSFRAME][3481860944] New Framework state: 3
2013-07-17 08:38:04.493: [ AGFW][3481860944] Agent Framework started successfully
2013-07-17 08:38:04.493: [ AGFW][1182374208] Agfw engine module has enabled...
2013-07-17 08:38:04.493: [CLSFRAME][1182374208] Module Enabling is complete
2013-07-17 08:38:04.493: [CLSFRAME][1182374208] New Framework state: 6
2013-07-17 08:38:04.493: [ AGFW][1182374208] Agent is started with userid: root , expected user: root
2013-07-17 08:38:04.493: [ AGFW][1182374208] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
2013-07-17 08:38:04.505: [ AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.crs.type] ID 8196:358
2013-07-17 08:38:04.506: [ AGFW][1182374208] Added new restype: ora.crs.type
2013-07-17 08:38:04.506: [ AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.crs.type] ID 8196:358
2013-07-17 08:38:04.506: [ AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.ctss.type] ID 8196:360
2013-07-17 08:38:04.506: [ AGFW][1182374208] Added new restype: ora.ctss.type
2013-07-17 08:38:04.507: [ AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.ctss.type] ID 8196:360
2013-07-17 08:38:04.516: [ AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.diskmon.type] ID 8196:362
2013-07-17 08:38:04.516: [ AGFW][1182374208] Added new restype: ora.diskmon.type
2013-07-17 08:38:04.517: [ AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.diskmon.type] ID 8196:362
2013-07-17 08:38:04.519: [ AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:364
2013-07-17 08:38:04.520: [ AGFW][1182374208] Added new restype: ora.drivers.acfs.type
2013-07-17 08:38:04.520: [ AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:364
2013-07-17 08:38:04.521: [ AGFW][1182374208] Agent received the message: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:366
2013-07-17 08:38:04.521: [ AGFW][1182374208] Added new resource: ora.diskmon 1 1 to the agfw
2013-07-17 08:38:04.522: [ AGFW][1182374208] Agent sending last reply for: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:366
2013-07-17 08:38:04.522: [ AGFW][1182374208] Agent received the message: RESOURCE_START[ora.diskmon 1 1] ID 4098:367
2013-07-17 08:38:04.522: [ AGFW][1182374208] Preparing START command for: ora.diskmon 1 1
2013-07-17 08:38:04.522: [ AGFW][1182374208] ora.diskmon 1 1 state changed from: UNKNOWN to: STARTING
2013-07-17 08:38:04.526: [ AGFW][1161394496] Executing command: start for resource: ora.diskmon 1 1
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] clsn_agent::start {
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DaemonAgent{
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] }DaemonAgent
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DiskmonAgent::DiskmonAgent {
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] InitAttrs {
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] __IS_HASD_AGENT=TRUE
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] }InitAttrs
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DiskmonAgent::DiskmonAgent }
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DiskmonAgent::start {
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Arg Value = -d
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Arg Value = -f
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Total Count of Environment Variables = 3
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Adding Environment Variables _ORA_AGENT_ACTION=TRUE
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Adding Environment Variables __IS_HASD_AGENT=
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Adding Environment variable from USR_ORA_ENV ORACLE_USER=oracle
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Utils:execCmd action = 1 flags = 5 ohome = (null) cmdname = diskmon.
2013-07-17 08:38:04.528: [ora.diskmon][1161394496] [start] getOracleHomeAttrib: oracle_home = /u01/app/product/11.2.0/oracle
2013-07-17 08:38:04.528: [ora.diskmon][1161394496] [start] Utils:execCmd Running the binary from /u01/app/product/11.2.0/oracle/bin/diskmon
2013-07-17 08:38:04.531: [CRSTIMER][1091324224] Timer Thread Starting.
2013-07-17 08:38:04.533: [ora.diskmon][1161394496] [start] execCmd ret = 0
2013-07-17 08:38:04.533: [ora.diskmon][1161394496] [start] }DaemonAgent::start
2013-07-17 08:38:10.534: [ora.diskmon][1161394496] [start] DiskmonAgent::connect {
2013-07-17 08:38:10.534: [ora.diskmon][1161394496] [start] Process::convertPidToString pid = 10319
2013-07-17 08:38:10.535: [ora.diskmon][1161394496] [start] DiskmonAgent::connect }
2013-07-17 08:38:10.535: [ora.diskmon][1161394496] [start] DiskmonAgent::start }
2013-07-17 08:38:10.535: [ora.diskmon][1161394496] [start] clsn_agent::start }
2013-07-17 08:38:10.535: [ AGFW][1161394496] Command: start for resource: ora.diskmon 1 1 completed with status: SUCCESS
2013-07-17 08:38:10.535: [ AGFW][1182374208] Agent sending reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:367
2013-07-17 08:38:10.537: [ AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:38:10.538: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
2013-07-17 08:38:10.538: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 0
2013-07-17 08:38:10.538: [ AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:38:10.538: [ AGFW][1182374208] ora.diskmon 1 1 state changed from: STARTING to: ONLINE
2013-07-17 08:38:10.538: [ AGFW][1182374208] Started implicit monitor for:ora.diskmon 1 1
2013-07-17 08:38:10.538: [ AGFW][1182374208] Agent sending last reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:367
2013-07-17 08:38:30.543: [ AGFW][1182374208] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:38:30.544: [ AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:38:30.544: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
2013-07-17 08:38:30.544: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 0
2013-07-17 08:38:30.545: [ AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:38:50.550: [ AGFW][1182374208] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:38:50.551: [ AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:38:50.551: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
2013-07-17 08:38:50.551: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 0
2013-07-17 08:38:50.551: [ AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:39:02.782: [ AGFW][1182374208] Agent received the message: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:547
2013-07-17 08:39:02.782: [ AGFW][1182374208] Preparing STOP command for: ora.diskmon 1 1
2013-07-17 08:39:02.782: [ AGFW][1182374208] ora.diskmon 1 1 state changed from: ONLINE to: STOPPING
2013-07-17 08:39:02.783: [ AGFW][1161394496] Executing command: stop for resource: ora.diskmon 1 1
2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] clsn_agent::stop {
2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] DiskmonAgent::stop {
2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] DiskmonAgent::stop }
2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] clsn_agent::stop }
2013-07-17 08:39:02.783: [ AGFW][1161394496] Command: stop for resource: ora.diskmon 1 1 completed with status: SUCCESS
2013-07-17 08:39:02.784: [ AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:39:02.784: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
2013-07-17 08:39:02.784: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 2
2013-07-17 08:39:02.784: [ AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: PLANNED_OFFLINE
2013-07-17 08:39:02.784: [ AGFW][1182374208] Agent sending reply for: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:547
2013-07-17 08:39:02.785: [ AGFW][1182374208] ora.diskmon 1 1 state changed from: STOPPING to: PLANNED_OFFLINE
2013-07-17 08:39:02.785: [ AGFW][1182374208] Agent sending last reply for: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:547
2013-07-17 08:39:02.785: [ AGFW][1182374208] Agent has no resources to be monitored.Sending suicide request.
2013-07-17 08:39:02.786: [ AGFW][1182374208] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:72
2013-07-17 08:39:02.789: [ AGFW][1182374208] Agent is commiting suicide.
2013-07-17 08:39:02.790: [ AGFW][1182374208] Agent is exiting with exit code: 1
2013-07-17 08:39:12.176: [ AGFW][1664629584] Starting the agent: /u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/
2013-07-17 08:39:12.176: [ AGENT][1664629584] Agent framework initialized, Process Id = 10581
2013-07-17 08:39:12.178: [ USRTHRD][1664629584] Utils::getCrsHome crsHome /u01/app/product/11.2.0/oracle
2013-07-17 08:39:12.178: [ USRTHRD][1664629584] Process::convertPidToString pid = 10581
2013-07-17 08:39:12.178: [ AGFW][1664629584] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
2013-07-17 08:39:12.178: [CLSFRAME][1664629584] Inited lsf context 0xd9309e0
2013-07-17 08:39:12.179: [CLSFRAME][1664629584] Initing CLS Framework messaging
2013-07-17 08:39:12.179: [CLSFRAME][1664629584] New Framework state: 2
2013-07-17 08:39:12.179: [CLSFRAME][1664629584] M2M is starting...
2013-07-17 08:39:12.180: [ CRSCOMM][1664629584] m_pClscCtx=0xd983bd0m_pUgblm=0xd987720
2013-07-17 08:39:12.180: [ CRSCOMM][1664629584] Starting send thread
2013-07-17 08:39:12.181: [ CRSCOMM][1115052352] clsIpc: sendWork thread started.
2013-07-17 08:39:12.181: [ CRSCOMM][1125542208] IPC Client thread started listening
2013-07-17 08:39:12.181: [ CRSCOMM][1125542208] init data sent from server
2013-07-17 08:39:12.181: [CLSFRAME][1664629584] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:vmorarac2
2013-07-17 08:39:12.181: [CLSFRAME][1664629584] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:vmorarac2
2013-07-17 08:39:12.182: [CLSFRAME][1664629584] Starting thread model named: MultiThread
2013-07-17 08:39:12.182: [CLSFRAME][1664629584] Starting thread model named: SingleThread
2013-07-17 08:39:12.182: [CLSFRAME][1664629584] Starting thread model named: SingleThreadT
2013-07-17 08:39:12.182: [CLSFRAME][1664629584] New Framework state: 3
2013-07-17 08:39:12.182: [ AGFW][1664629584] Agent Framework started successfully
2013-07-17 08:39:12.182: [ AGFW][1177991488] Agfw engine module has enabled...
2013-07-17 08:39:12.183: [CLSFRAME][1177991488] Module Enabling is complete
2013-07-17 08:39:12.183: [CLSFRAME][1177991488] New Framework state: 6
2013-07-17 08:39:12.183: [ AGFW][1177991488] Agent is started with userid: root , expected user: root
2013-07-17 08:39:12.183: [ AGFW][1177991488] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
2013-07-17 08:39:12.192: [ AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.crs.type] ID 8196:886
2013-07-17 08:39:12.192: [ AGFW][1177991488] Added new restype: ora.crs.type
2013-07-17 08:39:12.192: [ AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.crs.type] ID 8196:886
2013-07-17 08:39:12.198: [ AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.ctss.type] ID 8196:888
2013-07-17 08:39:12.198: [ AGFW][1177991488] Added new restype: ora.ctss.type
2013-07-17 08:39:12.199: [ AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.ctss.type] ID 8196:888
2013-07-17 08:39:12.204: [ AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.diskmon.type] ID 8196:890
2013-07-17 08:39:12.204: [ AGFW][1177991488] Added new restype: ora.diskmon.type
2013-07-17 08:39:12.204: [ AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.diskmon.type] ID 8196:890
2013-07-17 08:39:12.209: [ AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:892
2013-07-17 08:39:12.209: [ AGFW][1177991488] Added new restype: ora.drivers.acfs.type
2013-07-17 08:39:12.210: [ AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:892
2013-07-17 08:39:12.210: [ AGFW][1177991488] Agent received the message: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:894
2013-07-17 08:39:12.210: [ AGFW][1177991488] Added new resource: ora.diskmon 1 1 to the agfw
2013-07-17 08:39:12.210: [ AGFW][1177991488] Agent sending last reply for: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:894
2013-07-17 08:39:12.210: [ AGFW][1177991488] Agent received the message: RESOURCE_START[ora.diskmon 1 1] ID 4098:895
2013-07-17 08:39:12.211: [ AGFW][1177991488] Preparing START command for: ora.diskmon 1 1
2013-07-17 08:39:12.211: [ AGFW][1177991488] ora.diskmon 1 1 state changed from: UNKNOWN to: STARTING
2013-07-17 08:39:12.216: [ AGFW][1167501632] Executing command: start for resource: ora.diskmon 1 1
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] clsn_agent::start {
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] DaemonAgent{
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] }DaemonAgent
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] DiskmonAgent::DiskmonAgent {
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] InitAttrs {
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] __IS_HASD_AGENT=TRUE
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] }InitAttrs
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] DiskmonAgent::DiskmonAgent }
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] DiskmonAgent::start {
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Arg Value = -d
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Arg Value = -f
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Total Count of Environment Variables = 3
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Adding Environment Variables _ORA_AGENT_ACTION=TRUE
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Adding Environment Variables __IS_HASD_AGENT=
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Adding Environment variable from USR_ORA_ENV ORACLE_USER=oracle
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Utils:execCmd action = 1 flags = 5 ohome = (null) cmdname = diskmon.
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] getOracleHomeAttrib: oracle_home = /u01/app/product/11.2.0/oracle
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Utils:execCmd Running the binary from /u01/app/product/11.2.0/oracle/bin/diskmon
2013-07-17 08:39:12.220: [CRSTIMER][1198971200] Timer Thread Starting.
2013-07-17 08:39:12.220: [ora.diskmon][1167501632] [start] execCmd ret = 0
2013-07-17 08:39:12.220: [ora.diskmon][1167501632] [start] }DaemonAgent::start
2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] DiskmonAgent::connect {
2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] Process::convertPidToString pid = 10581
2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] DiskmonAgent::connect }
2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] DiskmonAgent::start }
2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] clsn_agent::start }
2013-07-17 08:39:18.222: [ AGFW][1167501632] Command: start for resource: ora.diskmon 1 1 completed with status: SUCCESS
2013-07-17 08:39:18.223: [ AGFW][1177991488] Agent sending reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:895
2013-07-17 08:39:18.223: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:39:18.223: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:39:18.224: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:39:18.224: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:39:18.225: [ AGFW][1177991488] ora.diskmon 1 1 state changed from: STARTING to: ONLINE
2013-07-17 08:39:18.225: [ AGFW][1177991488] Started implicit monitor for:ora.diskmon 1 1
2013-07-17 08:39:18.225: [ AGFW][1177991488] Agent sending last reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:895
2013-07-17 08:39:38.231: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:39:38.232: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:39:38.232: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:39:38.232: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:39:38.232: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:39:58.237: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:39:58.238: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:39:58.238: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:39:58.238: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:39:58.238: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:40:12.107: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:966
2013-07-17 08:40:18.243: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:40:18.244: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:40:18.244: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:40:18.244: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:40:18.244: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:40:38.250: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:40:38.251: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:40:38.251: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:40:38.251: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:40:38.251: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:40:42.116: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:982
2013-07-17 08:40:58.245: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:40:58.246: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:40:58.247: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:40:58.247: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:40:58.247: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:41:12.125: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:998
2013-07-17 08:41:18.252: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:41:18.252: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:41:18.253: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:41:18.253: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:41:18.253: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:41:38.259: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:41:38.260: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:41:38.260: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:41:38.260: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:41:38.260: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:41:58.255: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:41:58.256: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:41:58.256: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:41:58.256: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:41:58.256: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:42:12.134: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1026
2013-07-17 08:42:18.261: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:42:18.262: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:42:18.262: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:42:18.262: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:42:18.262: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:42:38.268: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:42:38.269: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:42:38.269: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:42:38.269: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:42:38.269: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:42:58.265: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:42:58.266: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:42:58.266: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:42:58.267: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:42:58.267: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:43:12.144: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1054
2013-07-17 08:43:18.272: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:43:18.272: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:43:18.273: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:43:18.273: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:43:18.273: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:43:38.278: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:43:38.278: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:43:38.279: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:43:38.279: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:43:38.279: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:43:42.154: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1070
2013-07-17 08:43:58.284: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:43:58.285: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:43:58.285: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:43:58.285: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:43:58.285: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:44:12.154: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1086
2013-07-17 08:44:18.291: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:44:18.292: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:44:18.292: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:44:18.292: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:44:18.292: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:44:38.296: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:44:38.297: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:44:38.297: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:44:38.297: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:44:38.297: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:44:42.163: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1102
2013-07-17 08:44:58.302: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:44:58.303: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:44:58.303: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:44:58.304: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:44:58.304: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:45:12.174: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1118
2013-07-17 08:45:18.309: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:45:18.309: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:45:18.310: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:45:18.310: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:45:18.310: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:45:38.315: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:45:38.316: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:45:38.316: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:45:38.316: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:45:38.316: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:45:42.183: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1134
2013-07-17 08:45:58.312: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:45:58.313: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:45:58.313: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:45:58.313: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:45:58.313: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:46:12.192: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1150
2013-07-17 08:46:18.318: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:46:18.319: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:46:18.319: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:46:18.319: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:46:18.319: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:46:38.325: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:46:38.326: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:46:38.326: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:46:38.326: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:46:38.326: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:46:42.203: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1166
2013-07-17 08:46:58.332: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:46:58.333: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:46:58.333: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:46:58.333: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:46:58.333: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:47:12.203: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1182
2013-07-17 08:47:18.338: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:47:18.339: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:47:18.339: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:47:18.339: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:47:18.340: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:47:38.345: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:47:38.345: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:47:38.346: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:47:38.346: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:47:38.346: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:47:42.211: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1198
2013-07-17 08:47:58.351: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:47:58.352: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:47:58.352: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:47:58.352: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:47:58.352: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:48:12.220: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1214
2013-07-17 08:48:18.358: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:48:18.359: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:48:18.359: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:48:18.359: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:48:18.359: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:48:38.365: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:48:38.366: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:48:38.366: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:48:38.366: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:48:38.366: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:48:42.230: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1230
2013-07-17 08:48:58.370: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:48:58.371: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:48:58.371: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:48:58.371: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:48:58.371: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:49:12.238: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1246
2013-07-17 08:49:18.377: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:49:18.378: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:49:18.378: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:49:18.379: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:49:18.379: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:49:23.262: [ AGFW][1177991488] Agent received the message: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:1310
2013-07-17 0 -
Ora.asm -init failed on second node root.sh
Hi All,
Installing Grid Infrastructure for a 11gr2 Cluster on two nodes Oracle Linux 5 + Vsware vSphere v4, shared disk on same host machine. When run root.sh, first node was success but the second node got following error message (actually the first node was cloned from the seoncd):
CRS-2672: Attempting to start 'ora.ctssd' on 'wandrac2'
Start action for octssd aborted
CRS-2676: Start of 'ora.ctssd' on 'wandrac2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'wandrac2'
CRS-2672: Attempting to start 'ora.asm' on 'wandrac2'
CRS-2676: Start of 'ora.drivers.acfs' on 'wandrac2' succeeded
CRS-2676: Start of 'ora.asm' on 'wandrac2' succeeded
CRS-2664: Resource 'ora.ctssd' is already running on 'wandrac2'
CRS-4000: Command Start failed, or completed with errors.
Command return code of 1 (256) from command: /orapp/racsl/11.2.0/bin/crsctl start resource ora.asm -init
Start of resource "ora.asm -init" failed
Failed to start ASM
Failed to start Oracle Clusterware stack
Thanks in advance for any information and helps,Hi,
I came across this error and I am about to start a fresh installation of the grid. (ealier one failed because it was unable to read the memory in rac2 )
Is there anything specific I can change before I start my installation.
PS - I didnt get what exactly is going on with the hosts file.
My files are as follows :
RAC1 - etc/hosts
[oracle@falcen6a ~]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
# Public
192.168.100.218 falcen6a.a.pri falcen6a
192.168.100.219 falcen6b.a.pri falcen6b
# Private
192.168.210.101 falcen6a-priv.a.pri falcen6a-priv
192.168.210.102 falcen6b-priv.a.pri falcen6b-priv
# Virtual
192.168.100.212 falcen6a-vip.a.pri falcen6a-vip
192.168.100.213 falcen6b-vip.a.pri falcen6b-vip
# SCAN
#192.168.100.208 falcen6-scan.a.pri falcen6-scan
#192.168.100.209 falcen6-scan.a.pri falcen6-scan
#192.168.100.210 falcen6-scan.a.pri falcen6-scan
on RAC2
[oracle@falcen6b ~]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
#Public
192.168.100.218 falcen6a.a.pri falcen6a
192.168.100.219 falcen6b.a.pri falcen6b
# Private
192.168.210.101 falcen6a-priv.a.pri falcen6a-priv
192.168.210.102 falcen6b-priv.a.pri falcen6b-priv
# Virtual
192.168.100.212 falcen6a-vip.a.pri falcen6a-vip
192.168.100.213 falcen6b-vip.a.pri falcen6b-vip
# SCAN
#192.168.100.208 falcen6-scan.a.pri falcen6-scan
#192.168.100.209 falcen6-scan.a.pri falcen6-scan
#192.168.100.210 falcen6-scan.a.pri falcen6-scan
Can someone please confirm this?? -
Inst_loc inventory pointer missing on second node in RAC
Hi,
I have installed Clusterware 11.1.0.6 on Windows 2003 on two nodes and I'd like to patch it to 11.1.0.7. However, when calling "opatch lsinventory -all" I noticed that it works on the first node (the node where the installation was performed), but doesn't work on the second node:
C:\product\11.1.0\crs\OPatch>opatch lsinventory -all
Invoking OPatch 11.1.0.6.0
Oracle Interim Patch Installer version 11.1.0.6.0
Copyright (c) 2007, Oracle Corporation. All rights reserved.
Oracle Home : C:\product\11.1.0\crs
Central Inventory : n/a
from : n/a
OPatch version : 11.1.0.6.0
OUI version : 11.1.0.6.0
OUI location : C:\product\11.1.0\crs\oui
Log file location : C:\product\11.1.0\crs\cfgtoollogs\opatch\opatch2010-11-28_10-22-19AM.log
OPatch cannot find a valid oraInst.loc file to locate Central Inventory.
OPatch failed with error code = 104
C:\product\11.1.0\crs\OPatch>I checked in the registry key "HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE" on both nodes and found out that the first node has an inventory pointer "inst_loc" wich points to "C:\Program Files\Oracle\Inventory", but on the second node this value is missing. If I manually set it on the second node, then "opatch lsinventory" works but I'm not sure if it's ok to set it manually?
I also checked the documentation and here http://download.oracle.com/docs/cd/B28359_01/em.111/b31207/oui5_cluster_environment.htm#OUICG267 it says: "After you click Next, the Oracle Universal Installer checks whether the remote inventories are set. If they are not set, the Oracle Universal Installer sets up the remote inventories by setting registry keys." So I suppose the registry key should exist also on the second node?
Thanks in advance for any answers.
Regards,
JureHi, mmm ... looks like remote operations fail during the installation process. Possible reasons for this error could be either a missing oraInst.loc file or permission issues with oraInst.loc file. Ensure user have read/write priviligies on orainst.loc file as well as on the actual path of orainventory location. If you could not fix the issue and if you know the inventory location you may want try following solution: "opatch apply -invPtrLoc C:\mypath\mypath\oraInst.loc" where "mypath" should be replaced by your windows locations. Also review you have all the oracle binaries in the 2nd. node.
Edited by: Jose Valerio on Nov 28, 2010 2:21 PM
Edited by: Jose Valerio on Nov 28, 2010 2:21 PM
Maybe you are looking for
-
Network based and a-gps, data use tests
i was trying to find this kind of info, but only found others asking. i hope this is useful info for others, especially while roaming. skip to the bottom for conclusion, this is the process i took: i called nokia tech support, their phone support his
-
My iphone 4s is always losing signal,help plz
My iphone 4s is always losin signal.its in places were i always got a signal with my old phone and everyone seems to ave a good signal. All thru the day 1 min il have full signal to no service,any help plz
-
How do I get the numbers ipad app to plot a graph of dates against values, where the most recent date is on the right of the graph, but the most recent date is at the top rather than bottom of the rows in the table? Also how can it be a line graph wi
-
Hi, The below is an example of 3 tables for which i want to cluster. I have the following structure of the 3 tables: Table A Primary key -> referenced by Table B Table A & B Primary key -> referenced by Table C I have done the following: 1. This is t
-
Hi all, I'm a functional newbie to ESS/MSS and would like to know how technical/functional both the packages are in terms of configuration. I understand that though ESS, employees manage their personal data and through MSS, managers can have better c