Runcluvfy.sh stage pre fails with node reachability on 1 node only

Question

Having a frustrating problem. 2 node RAC system on RHEL 5.2 installing 11.2.0.1 grid/clusterware. Performing the following pre check command from node 1:
./runcluvfy.sh stage -pre crsinst -n node1,node2 -verboseI'm getting the following error and it cannot write the trace information
[grid@node1 grid]$ sudo chmod -R 777 /tmp
[grid@node1 grid]$ ./runcluvfy.sh stage -pre crsinst -n node1,node2 -verbose
WARNING:
Could not access or create trace file path "/tmp/bootstrap/cv/log". Trace information could not be collected
Performing pre-checks for cluster services setup
Checking node reachability...
node1.mydomain.com: node1.mydomain.com
Check: Node reachability from node "null"
  Destination Node                      Reachable?
  node2                       no
  node1                       no
Result: Node reachability check failed from node "null"
ERROR:
Unable to reach any of the nodes
Verification cannot proceed
Pre-check for cluster services setup was unsuccessful on all the nodes.
[grid@node1 grid]$
[grid@node1 grid]$ echo $CV_DESTLOC
/home/grid/software/grid/11gr2/gridI've verified the following:
1) there is user equivalence between the nodes for user grid
2) /tmp is read/writable by user grid on both nodes
3) Setting the CV_DESTLOC appears to do nothing - it seems to go back to wanting to write to /tmp
4) ./runcluvfy comp nodecon -n node1,node2-verbose succeeds no problem
And the weirdest thing of all, when I run ./runcluvfy.sh stage -pre crsinst -n node1,node2 -verbose from node 2, it succeeds without errors.
What am I missing? And TIA..

Answer 1

I made a copy of the runcluvfy.sh and commented out all rm -rf commands so that it would at least save the trace files. Re-ran, and the following trace output - not entirely helpful to me, but any gurus out there see anything?
[main] [ 2010-04-20 15:48:38.275 CDT ] [TaskNodeConnectivity.performTask:354]  _nw_:Performing Node Reachability verification task...
[main] [ 2010-04-20 15:48:38.282 CDT ] [ResultSet.traceResultSet:341]
Target ResultSet BEFORE Upload===>
        Overall Status->UNKNOWN
[main] [ 2010-04-20 15:48:38.283 CDT ] [ResultSet.traceResultSet:341]
Source ResultSet ===>
        Overall Status->OPERATION_FAILED
        node2-->OPERATION_FAILED
        node1-->OPERATION_FAILED
[main] [ 2010-04-20 15:48:38.283 CDT ] [ResultSet.traceResultSet:341]
Target ResultSet AFTER Upload===>
        Overall Status->OPERATION_FAILED
        node2-->OPERATION_FAILED
        node1-->OPERATION_FAILED
[main] [ 2010-04-20 15:48:38.284 CDT ] [ResultSet.getSuccNodes:556]  Checking for Success nodes from the total list of nodes in the resultset
[main] [ 2010-04-20 15:48:38.284 CDT ] [ReportUtil.printReportFooter:1553]  stageMsgID: 8302
[main] [ 2010-04-20 15:48:38.284 CDT ] [CluvfyDriver.main:299]  ==== cluvfy exiting normally.I'm still baffled why the precheck is successful from the second node. And, in fact, all other cluvfy checks that I've run succeed form both nodes.