INS-20802 Oracle Private Interconnect Configuration Assistant failed

Question

Thought I would post what information I've gathered, after facing this error during install of RAC Grid Infrastructure 11.2.0.1 on Red Hat Enterprise Linux Server release 5.5 64-bit, as Oracle Support is once again unable to help. Maybe this will save someone else some time and the aggravation of dealing with lousy Oracle Support.
The error occurs after root.sh has successfully completed on all nodes. Oracle Net Configuration Assistant runs successfully, then Oracle Private Interconnect Configuration Assistant launches and subsequently fails with the following.
[INS-20802] Oracle Private Interconnect Configuration Assistant failed.
/u01/app/oraInventory/logs/installActions2010-12-13_01-26-10PM.log
INFO: Starting 'Oracle Private Interconnect Configuration Assistant'
INFO: Starting 'Oracle Private Interconnect Configuration Assistant'
INFO: PRIF-26: Error in update the profiles in the cluster
INFO:
WARNING:
INFO: Completed Plugin named: Oracle Private Interconnect Configuration Assistant
INFO: Oracle Private Interconnect Configuration Assistant failed.
INFO: Oracle Private Interconnect Configuration Assistant failed.
I was able to find another error that coincides with the PRIF-26 error: "CRS-2324:Error(s) occurred while trying to push GPnP profile. State may be inconsistent."
I was also able to duplicate the PRIF-26 error by trying to add a non-existent network interface via oifcfg:
./oifcfg setif -global jjj1/192.167.1.0:cluster_interconnect
PRIF-26: Error in update the profiles in the cluster
My best guess is the Oracle Private Interconnect Configuration Assistant makes a call to oifcfg. When oifcfg makes an update or adds a public/private interface, some XML files are also updated or maybe cross-referenced. These files are located here: <grid_home>/gpnp/<host>/profiles/peer
Any updates/changes/addtions to the private or public interfaces include changes for the Grid Plug-n-Play component, which uses the XML files. If the interface name is not contained in the XML files, my best guess is that triggers the "CRS-2324:Error(s) occurred while trying to push GPnP profile. State may be inconsistent.
I verified everything was configured correctly; the cluster verification utility reported everything was ok. I also ran the cluster verifcation utility against the GP-nP:
./cluvfy comp gpnp -verbose
I also reviewed the public and private interfaces via oifcfg and they are correct:
[oracle@ryoradb1 bin]$ ./oifcfg getif -global
eth0 10.10.2.0 global public
eth1 192.167.1.0 global cluster_interconnect
[oracle@ryoradb1 bin]$ ./oifcfg iflist -p
eth0 10.10.2.0 PRIVATE
eth1 192.167.1.0 UNKNOWN
My conclusion is the environment is configured correctly, in spite of the error generated by the Oracle Private Configuration Assistant.

Answer 1

I understand that you have installed 11.2.0.1 not 11.2.0.2 because multicating must be enabled if you have installed 11.2.0.2 and you may face these sort of problems because cluster nodes would not be able to communicate with each other.
Please check ocssd.log especially on the first node because this file will give more inforamtion as to why first node is not able to push GPnP file. As you have executed cluvfy to confirm the GPnP but to confirm whether GPnP profile is accurate to just narrow down the problem, I would suggest you to try to start cluster in exclusive mode so that you can sure that GPnP profile is good.
Shutdown CRS on all nodes, if there is any hung processes then kill them before executing the fllowing command to start cluster in exclusive mode.
$GRID_HOME/bin/crsctl start crs -excl
If you are able to start cluster in exclusive then it is sure that GPnP is correct and then next step would be to verify the private network.
See how you goes.
FYI, I was recently analyzing the same sort of problems where cluster was not able to access GPnP profile and finally I found issues on my private network. Customer had enabled IGMP snooping, which was avoiding multicast communication over the private network but it was 11.2.0.2, which is not the case here.
Harish Kumar
http://www.oraxperts.com