Failing all pre-requisite checks for 11.2.0.2.0 RAC on HP-UX Itanium 64

Question

Hi Guys,
For Typical OR Advanced Grid Installation (GUI), all pre-requisites are failing on both nodes, like memory, swap, node reachability, groups, user & lot more.....
But when I try to check Pre-requisite from command line, everything seems to be ok.
This is what the output...
$ ./runcluvfy.sh stage -pre crsinst -n sph1erp,sph2erp -fixup -verbose
Performing pre-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "sph1erp"
Destination Node Reachable?
sph1erp yes
sph2erp yes
Result: Node reachability check passed from node "sph1erp"
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
sph2erp passed
sph1erp passed
Result: User equivalence check passed for user "oracle"
Checking node connectivity...
Checking hosts config file...
Node Name Status Comment
sph2erp passed
sph1erp passed
Verification of the hosts config file successful
Interface information for node "sph2erp"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
lan2 10.10.16.51 10.10.16.48 10.10.16.51 10.10.1.12 78:AC:C0:8A:07:76 1500
lan900 10.10.1.174 10.10.0.0 10.10.1.174 10.10.1.12 78:AC:C0:8A:07:6E 1500
Interface information for node "sph1erp"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
lan2 10.10.16.50 10.10.16.48 10.10.16.50 10.10.1.12 3C:4A:92:48:71:BE 1500
lan900 10.10.1.173 10.10.0.0 10.10.1.173 10.10.1.12 3C:4A:92:48:71:B6 1500
Check: Node connectivity of subnet "10.10.16.48"
Source Destination Connected?
sph2erp[10.10.16.51] sph1erp[10.10.16.50] yes
Result: Node connectivity passed for subnet "10.10.16.48" with node(s) sph2erp,sph1erp
Check: TCP connectivity of subnet "10.10.16.48"
Source Destination Connected?
sph1erp:10.10.16.50 sph2erp:10.10.16.51 passed
Result: TCP connectivity check passed for subnet "10.10.16.48"
Check: Node connectivity of subnet "10.10.0.0"
Source Destination Connected?
sph2erp[10.10.1.174] sph1erp[10.10.1.173] yes
Result: Node connectivity passed for subnet "10.10.0.0" with node(s) sph2erp,sph1erp
Check: TCP connectivity of subnet "10.10.0.0"
Source Destination Connected?
sph1erp:10.10.1.173 sph2erp:10.10.1.174 passed
Result: TCP connectivity check passed for subnet "10.10.0.0"
Interfaces found on subnet "10.10.0.0" that are likely candidates for VIP are:
sph2erp lan900:10.10.1.174
sph1erp lan900:10.10.1.173
Interfaces found on subnet "10.10.16.48" that are likely candidates for a private interconnect are:
sph2erp lan2:10.10.16.51
sph1erp lan2:10.10.16.50
Result: Node connectivity check passed
Check: Total memory
Node Name Available Required Comment
sph2erp 63.9GB (6.7004024E7KB) 4GB (4194304.0KB) passed
sph1erp 63.9GB (6.7004024E7KB) 4GB (4194304.0KB) passed
Result: Total memory check passed
Check: Available memory
Node Name Available Required Comment
sph2erp 53.3556GB (5.5947372E7KB) 50MB (51200.0KB) passed
sph1erp 52.2323GB (5.47695E7KB) 50MB (51200.0KB) passed
Result: Available memory check passed
Check: Swap space
Node Name Available Required Comment
sph2erp 188.7773GB (1.97947352E8KB) 4GB (4194304.0KB) passed
sph1erp 188.7773GB (1.97947352E8KB) 4GB (4194304.0KB) passed
Result: Swap space check passed
Check: Free disk space for "sph2erp:/var/tmp/"
Path Node Name Mount point Available Required Comment
/var/tmp/ sph2erp /var 13.8074GB 1GB passed
Result: Free disk space check passed for "sph2erp:/var/tmp/"
Check: Free disk space for "sph1erp:/var/tmp/"
Path Node Name Mount point Available Required Comment
/var/tmp/ sph1erp /var 13.8158GB 1GB passed
Result: Free disk space check passed for "sph1erp:/var/tmp/"
Check: User existence for "oracle"
Node Name Status Comment
sph2erp exists(199) passed
sph1erp exists(199) passed
Checking for multiple users with UID value 199
Result: Check for multiple users with UID value 199 passed
Result: User existence check passed for "oracle"
Check: Group existence for "oinstall"
Node Name Status Comment
sph2erp exists passed
sph1erp exists passed
Result: Group existence check passed for "oinstall"
Check: Group existence for "dba"
Node Name Status Comment
sph2erp exists passed
sph1erp exists passed
Result: Group existence check passed for "dba"
Check: Membership of user "oracle" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Comment
sph2erp yes yes yes yes passed
sph1erp yes yes yes yes passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed
Check: Membership of user "oracle" in group "dba"
Node Name User Exists Group Exists User in Group Comment
sph2erp yes yes yes passed
sph1erp yes yes yes passed
Result: Membership check for user "oracle" in group "dba" passed
Check: Run level
Node Name run level Required Comment
sph2erp 3 3 passed
sph1erp 3 3 passed
Result: Run level check passed
Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Comment
sph2erp hard 63488 63488 passed
sph1erp hard 63488 63488 passed
Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Comment
sph2erp soft 1024 1024 passed
sph1erp soft 63488 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes"
Node Name Type Available Required Comment
sph2erp hard 3687 3686 passed
sph1erp hard 3687 3686 passed
Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes"
Node Name Type Available Required Comment
sph2erp soft 3687 2047 passed
sph1erp soft 3687 2047 passed
Result: Soft limits check passed for "maximum user processes"
Check: System architecture
Node Name Available Required Comment
sph2erp ia64 ia64 passed
sph1erp ia64 ia64 passed
Result: System architecture check passed
Check: Kernel version
Node Name Available Required Comment
sph2erp HP-UX B.11.31 B.11.31 passed
sph1erp HP-UX B.11.31 B.11.31 passed
Result: Kernel version check passed
Check: Kernel parameter for "ksi_alloc_max"
Node Name Configured Required Comment
sph2erp 32768 32768 passed
sph1erp 32768 32768 passed
Result: Kernel parameter check passed for "ksi_alloc_max"
Check: Kernel parameter for "executable_stack"
Node Name Configured Required Comment
sph2erp 0 0 passed
sph1erp 0 0 passed
Result: Kernel parameter check passed for "executable_stack"
Check: Kernel parameter for "max_thread_proc"
Node Name Configured Required Comment
sph2erp 1024 1024 passed
sph1erp 1024 1024 passed
Result: Kernel parameter check passed for "max_thread_proc"
Check: Kernel parameter for "maxdsiz"
Node Name Configured Required Comment
sph2erp 1073741824 1073741824 passed
sph1erp 1073741824 1073741824 passed
Result: Kernel parameter check passed for "maxdsiz"
Check: Kernel parameter for "maxdsiz_64bit"
Node Name Configured Required Comment
sph2erp 2147483648 2147483648 passed
sph1erp 2147483648 2147483648 passed
Result: Kernel parameter check passed for "maxdsiz_64bit"
Check: Kernel parameter for "maxssiz"
Node Name Configured Required Comment
sph2erp 134217728 134217728 passed
sph1erp 134217728 134217728 passed
Result: Kernel parameter check passed for "maxssiz"
Check: Kernel parameter for "maxssiz_64bit"
Node Name Configured Required Comment
sph2erp 1073741824 1073741824 passed
sph1erp 1073741824 1073741824 passed
Result: Kernel parameter check passed for "maxssiz_64bit"
Check: Kernel parameter for "maxuprc"
Node Name Configured Required Comment
sph2erp 3686 3686 passed
sph1erp 3686 3686 passed
Result: Kernel parameter check passed for "maxuprc"
Check: Kernel parameter for "msgmni"
Node Name Configured Required Comment
sph2erp 4096 4096 passed
sph1erp 4096 4096 passed
Result: Kernel parameter check passed for "msgmni"
Check: Kernel parameter for "msgtql"
Node Name Configured Required Comment
sph2erp 4096 4096 passed
sph1erp 4096 4096 passed
Result: Kernel parameter check passed for "msgtql"
Check: Kernel parameter for "ncsize"
Node Name Configured Required Comment
sph2erp 35840 35840 passed
sph1erp 35840 35840 passed
Result: Kernel parameter check passed for "ncsize"
Check: Kernel parameter for "shmmax"
Node Name Configured Required Comment
sph2erp 1073741824 1073741824 passed
sph1erp 1073741824 1073741824 passed
Result: Kernel parameter check passed for "shmmax"
Check: Kernel parameter for "shmmni"
Node Name Configured Required Comment
sph2erp 4096 4096 passed
sph1erp 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"
Check: Kernel parameter for "shmseg"
Node Name Configured Required Comment
sph2erp 512 512 passed
sph1erp 512 512 passed
Result: Kernel parameter check passed for "shmseg"
Check: Kernel parameter for "tcp_smallest_anon_port"
Node Name Configured Required Comment
sph2erp 9000 9000 passed
sph1erp 9000 9000 passed
Result: Kernel parameter check passed for "tcp_smallest_anon_port"
Check: Kernel parameter for "tcp_largest_anon_port"
Node Name Configured Required Comment
sph2erp 65500 65500 passed
sph1erp 65500 65500 passed
Result: Kernel parameter check passed for "tcp_largest_anon_port"
Check: Kernel parameter for "udp_smallest_anon_port"
Node Name Configured Required Comment
sph2erp 9000 9000 passed
sph1erp 9000 9000 passed
Result: Kernel parameter check passed for "udp_smallest_anon_port"
Check: Kernel parameter for "udp_largest_anon_port"
Node Name Configured Required Comment
sph2erp 65500 65500 passed
sph1erp 65500 65500 passed
Result: Kernel parameter check passed for "udp_largest_anon_port"
Check: Package existence for "OS-Core-B.11.31( ia64)"
Node Name Available Required Comment
sph2erp OS-Core-B.11.31-0 OS-Core-B.11.31( ia64) passed
sph1erp OS-Core-B.11.31-0 OS-Core-B.11.31( ia64) passed
Result: Package existence check passed for "OS-Core-B.11.31( ia64)"
Check: Operating system patch for "Patch PHKL_38938"
Node Name Applied Required Comment
sph2erp Patch PHKL_39646 Patch PHKL_38938 passed
sph1erp Patch PHKL_39646 Patch PHKL_38938 passed
Result: Operating system patch check passed for "Patch PHKL_38938"
Check: Operating system patch for "Patch PHKL_39351"
Node Name Applied Required Comment
sph2erp Patch PHKL_40207 Patch PHKL_39351 passed
sph1erp Patch PHKL_40207 Patch PHKL_39351 passed
Result: Operating system patch check passed for "Patch PHKL_39351"
Check: Operating system patch for "Patch PHSS_36354"
Node Name Applied Required Comment
sph2erp Patch PHSS_40546 Patch PHSS_36354 passed
sph1erp Patch PHSS_40546 Patch PHSS_36354 passed
Result: Operating system patch check passed for "Patch PHSS_36354"
Check: Operating system patch for "Patch PHSS_37042"
Node Name Applied Required Comment
sph2erp Patch PHSS_37042 Patch PHSS_37042 passed
sph1erp Patch PHSS_37042 Patch PHSS_37042 passed
Result: Operating system patch check passed for "Patch PHSS_37042"
Check: Operating system patch for "Patch PHSS_37959"
Node Name Applied Required Comment
sph2erp Patch PHSS_40804 Patch PHSS_37959 passed
sph1erp Patch PHSS_40804 Patch PHSS_37959 passed
Result: Operating system patch check passed for "Patch PHSS_37959"
Check: Operating system patch for "Patch PHSS_39094"
Node Name Applied Required Comment
sph2erp Patch PHSS_40538 Patch PHSS_39094 passed
sph1erp Patch PHSS_40538 Patch PHSS_39094 passed
Result: Operating system patch check passed for "Patch PHSS_39094"
Check: Operating system patch for "Patch PHSS_39100"
Node Name Applied Required Comment
sph2erp Patch PHSS_40540 Patch PHSS_39100 passed
sph1erp Patch PHSS_40540 Patch PHSS_39100 passed
Result: Operating system patch check passed for "Patch PHSS_39100"
Check: Operating system patch for "Patch PHSS_39102"
Node Name Applied Required Comment
sph2erp Patch PHSS_41496 Patch PHSS_39102 passed
sph1erp Patch PHSS_41496 Patch PHSS_39102 passed
Result: Operating system patch check passed for "Patch PHSS_39102"
Check: Operating system patch for "Patch PHSS_38141"
Node Name Applied Required Comment
sph2erp Patch PHSS_38141 Patch PHSS_38141 passed
sph1erp Patch PHSS_38141 Patch PHSS_38141 passed
Result: Operating system patch check passed for "Patch PHSS_38141"
Check: Operating system patch for "Patch PHCO_40381"
Node Name Applied Required Comment
sph2erp Patch PHCO_40381 Patch PHCO_40381 passed
sph1erp Patch PHCO_40381 Patch PHCO_40381 passed
Result: Operating system patch check passed for "Patch PHCO_40381"
Check: Operating system patch for "Patch PHKL_38038"
Node Name Applied Required Comment
sph2erp Patch PHKL_41005 Patch PHKL_38038 passed
sph1erp Patch PHKL_41005 Patch PHKL_38038 passed
Result: Operating system patch check passed for "Patch PHKL_38038"
Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed
Check: Current group ID
Result: Current group ID check passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Checking Core file name pattern consistency...
Core file name pattern consistency check passed.
Checking to make sure user "oracle" is not in "root" group
Node Name Status Comment
sph2erp does not exist passed
sph1erp does not exist passed
Result: User "oracle" is not part of "root" group. Check passed
Check default user file creation mask
Node Name Available Required Comment
sph2erp 022 0022 passed
sph1erp 022 0022 passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes
Node Name Status
sph2erp passed
sph1erp passed
The DNS response time for an unreachable node is within acceptable limit on all nodes
File "/etc/resolv.conf" is consistent across nodes
Check: Time zone consistency
Result: Time zone consistency check passed
Checking settings of device file "/dev/async"
Node Name Available Comment
sph2erp yes failed (incorrect setting for minor number.)
sph1erp yes failed (incorrect setting for minor number.)
Result: Check for settings of device file "/dev/async" failed.
Starting check for The SSH LoginGraceTime setting ...
WARNING:
PRVE-0038 : The SSH LoginGraceTime setting on node "sph2erp" may result in users being disconnected before login is completed
PRVE-0038 : The SSH LoginGraceTime setting on node "sph1erp" may result in users being disconnected before login is completed
Check for The SSH LoginGraceTime setting passed
Fixup information has been generated for following node(s):
sph2erp,sph1erp
Please run the following script on each node as "root" user to execute the fixups:
'/tmp/CVU_11.2.0.2.0_oracle/runfixup.sh'
Pre-check for cluster services setup was unsuccessful on all the nodes.
What could be the issue ????
Any help would be appreciated...
Regards,
Manish

Answer 1

Started with Grid Installation. Copying Software to remote node & linking libraries were successfully without any issue (upto 76%). But got issue while executing root.sh on Node1
sph1erp:/oracle/11.2.0/grid #sh root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/11.2.0/grid
Enter the full pathname of the local bin directory: [usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'sys'..
Operation successful.
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'sph1erp'
CRS-2676: Start of 'ora.mdnsd' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'sph1erp'
CRS-2676: Start of 'ora.gpnpd' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'sph1erp'
CRS-2672: Attempting to start 'ora.gipcd' on 'sph1erp'
CRS-2676: Start of 'ora.gipcd' on 'sph1erp' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'sph1erp'
CRS-2672: Attempting to start 'ora.diskmon' on 'sph1erp'
CRS-2676: Start of 'ora.diskmon' on 'sph1erp' succeeded
CRS-2676: Start of 'ora.cssd' on 'sph1erp' succeeded
ASM created and started successfully.
Disk Group OCRVOTE created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'sys'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk ab847ed2b4f04f2dbfb875226d2bb194.
Successful addition of voting disk 85c05a5b30384f8dbff48cc069de7a7c.
Successful addition of voting disk 649196fbdd614f9cbf26a9a0e6670a6e.
Successful addition of voting disk 8815dfcee2e64f64bf00b9c76626ab41.
Successful addition of voting disk 8ce55fe5534f4f77bfa9f54187592707.
Successfully replaced voting disk group with +OCRVOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
1. ONLINE ab847ed2b4f04f2dbfb875226d2bb194 (/dev/oracle/ocrvote1) [OCRVOTE]
2. ONLINE 85c05a5b30384f8dbff48cc069de7a7c (/dev/oracle/ocrvote2) [OCRVOTE]
3. ONLINE 649196fbdd614f9cbf26a9a0e6670a6e (/dev/oracle/ocrvote3) [OCRVOTE]
4. ONLINE 8815dfcee2e64f64bf00b9c76626ab41 (/dev/oracle/ocrvote4) [OCRVOTE]
5. ONLINE 8ce55fe5534f4f77bfa9f54187592707 (/dev/oracle/ocrvote5) [OCRVOTE]
Located 5 voting disk(s).
Start of resource "ora.cluster_interconnect.haip" failed
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'sph1erp'
CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error:
Start action for HAIP aborted
CRS-2674: Start of 'ora.cluster_interconnect.haip' on 'sph1erp' failed
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'sph1erp'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'sph1erp' succeeded
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Clusterware stack
Failed to start High Availability IP at /oracle/11.2.0/grid/crs/install/crsconfig_lib.pm line 1046.
/oracle/11.2.0/grid/perl/bin/perl -I/oracle/11.2.0/grid/perl/lib -I/oracle/11.2.0/grid/crs/install /oracle/11.2.0/grid/crs/install/rootcrs.pl execution failed
sph1erp:/oracle/11.2.0/grid #
Last few lines from CRS Log for node 1, where error came
[ctssd(6467)]CRS-2401:The Cluster Time Synchronization Service started on host sph1erp.
2011-02-25 23:04:16.491
[oracle/11.2.0/grid/bin/orarootagent.bin(6423)]CRS-5818:Aborted command 'start for resource: ora.cluster_interconnect.haip 1 1' for resource 'ora.cluster_int
erconnect.haip'. Details at (:CRSAGF00113:) {0:0:178} in */oracle/11.2.0/grid/log/sph1erp/agent/ohasd/orarootagent_root/orarootagent_root.log.*
2011-02-25 23:04:20.521
[ohasd(5513)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:178} in
*/oracle/11.2.0/grid/log/sph1erp/ohasd/ohasd.log.*
Few lines from */oracle/11.2.0/grid/log/sph1erp/agent/ohasd/orarootagent_root/orarootagent_root.log.*
=====================================================================================================
2011-02-25 23:04:16.823: [ USRTHRD][16] {0:0:178} Starting Probe for ip 169.254.74.54
2011-02-25 23:04:16.823: [ USRTHRD][16] {0:0:178} Transitioning to Probe State
2011-02-25 23:04:17.177: [ USRTHRD][15] {0:0:178} [NetHAMain] thread stopping
2011-02-25 23:04:17.177: [ USRTHRD][15] {0:0:178} Thread:[NetHAMain]isRunning is reset to false here
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} Thread:[NetHAMain]stop }
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} thread cleaning up
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} pausing thread
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} posting thread
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop {
2011-02-25 23:04:17.645: [ USRTHRD][16] {0:0:178} [NetHAWork] thread stopping
2011-02-25 23:04:17.645: [ USRTHRD][16] {0:0:178} Thread:[NetHAWork]isRunning is reset to false here
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop }
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop {
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop }
2011-02-25 23:04:17.891: [ora.cluster_interconnect.haip][12] {0:0:178} [start] Start of HAIP aborted
2011-02-25 23:04:17.892: [   AGENT][12] {0:0:178} UserErrorException: Locale is
2011-02-25 23:04:17.893: [ora.cluster_interconnect.haip][12] {0:0:178} [start] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error:
Start action for HAIP aborted
2011-02-25 23:04:17.893: [    AGFW][12] {0:0:178} sending status msg [CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the foll
owing error:
Start action for HAIP aborted
] for start for resource: ora.cluster_interconnect.haip 1 1
2011-02-25 23:04:17.893: [ora.cluster_interconnect.haip][12] {0:0:178} [start] clsn_agent::start }
2011-02-25 23:04:17.894: [    AGFW][10] {0:0:178} Agent sending reply for: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:661
2011-02-25 23:04:18.552: [ora.diskmon][12] {0:0:154} [check] DiskmonAgent::check {
2011-02-25 23:04:18.552: [ora.diskmon][12] {0:0:154} [check] DiskmonAgent::check } - 0
2011-02-25 23:04:19.573: [    AGFW][10] {0:0:154} Agent received the message: AGENT_HB[Engine] ID 12293:669
2011-02-25 23:04:20.510: [ora.cluster_interconnect.haip][18] {0:0:178} [start] got lock
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] tryActionLock }
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] abort }
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] clsn_agent::abort }
2011-02-25 23:04:20.511: [    AGFW][18] {0:0:178} Command: start for resource: ora.cluster_interconnect.haip 1 1 completed with status: TIMEDOUT
2011-02-25 23:04:20.512: [ora.cluster_interconnect.haip][8] {0:0:178} [check] NetworkAgent::init enter {
2011-02-25 23:04:20.513: [ora.cluster_interconnect.haip][8] {0:0:178} [check] NetworkAgent::init exit }
2011-02-25 23:04:20.517: [    AGFW][10] {0:0:178} Agent sending reply for: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:661
2011-02-25 23:04:20.519: [ USRTHRD][8] {0:0:178} Ocr Context init default level 23886304
2011-02-25 23:04:20.519: [ default][8]clsvactversion:4: Retrieving Active Version from local storage.
[   CLWAL][8]clsw_Initialize: OLR initlevel [70000]
Few lines from */oracle/11.2.0/grid/log/sph1erp/ohasd/ohasd.log.*
=====================================================================================================
2011-02-25 23:04:21.627: [UiServer][30] {0:0:180} Done for ctx=6000000002604ce0
2011-02-25 23:04:21.642: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:26.139: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:26.139: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:26.973: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:26.973: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:26.992: [UiServer][30] {0:0:181} processMessage called
2011-02-25 23:04:26.993: [UiServer][30] {0:0:181} Sending message to PE. ctx= 6000000001b440f0
2011-02-25 23:04:26.993: [UiServer][30] {0:0:181} Sending command to PE: 67
2011-02-25 23:04:26.994: [   CRSPE][29] {0:0:181} Processing PE command id=173. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:26.997: [UiServer][30] {0:0:181} Done for ctx=6000000001b440f0
2011-02-25 23:04:27.012: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:31.135: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:31.135: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:32.318: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:32.318: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:32.332: [UiServer][30] {0:0:182} processMessage called
2011-02-25 23:04:32.333: [UiServer][30] {0:0:182} Sending message to PE. ctx= 6000000001b45ef0
2011-02-25 23:04:32.333: [UiServer][30] {0:0:182} Sending command to PE: 68
2011-02-25 23:04:32.334: [   CRSPE][29] {0:0:182} Processing PE command id=174. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:32.338: [UiServer][30] {0:0:182} Done for ctx=6000000001b45ef0
2011-02-25 23:04:32.352: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:36.155: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:36.155: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:37.683: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:37.683: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:37.702: [UiServer][30] {0:0:183} processMessage called
2011-02-25 23:04:37.703: [UiServer][30] {0:0:183} Sending message to PE. ctx= 6000000002604ce0
2011-02-25 23:04:37.703: [UiServer][30] {0:0:183} Sending command to PE: 69
2011-02-25 23:04:37.704: [   CRSPE][29] {0:0:183} Processing PE command id=175. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:37.708: [UiServer][30] {0:0:183} Done for ctx=6000000002604ce0
2011-02-25 23:04:37.722: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:41.156: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:41.156: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
What could be the issue ????
Experts Please help me. Doing setup for the PRoduction Env...
Do response ASAP...... Thanks
Regards,
Manish