Stalled Grid 11GR2 installation at 65% - Performing Remote Operations

Question

Hello everyone,
I have some issues with the instalation of the 11G R2 Grid...
Passed all the prereq checks, everything working but when reaching the - Performing Remote Operations stage it freezes.
Checked the errors and when reaching that stage an error is dumped in the oraInstall.err file:
Exception in thread "Install API Thread" java.lang.NullPointerException
at oracle.cluster.deployment.ractrans.ClientHandlerSupervisor.threadCleanup(ClientHandlerSupervisor.java:926)
at oracle.cluster.deployment.ractrans.RACTransfer.cleanup(RACTransfer.java:1749)
at oracle.cluster.deployment.ractrans.RACTransfer.transferDirStructureToNodes(RACTransfer.java:746)
at oracle.cluster.deployment.ractrans.RACTransfer.transferDirToNodes(RACTransfer.java:252)
at oracle.ops.mgmt.cluster.ClusterCmd.transferDirToNodes(ClusterCmd.java:3103)
at oracle.ops.mgmt.cluster.ClusterCmd.transferDirToNodes(ClusterCmd.java:3022)
at oracle.sysman.oii.oiip.oiipg.OiipgClusterOps.transferDirToNodes(OiipgClusterOps.java:947)
at oracle.sysman.oii.oiif.oiifw.OiifwClusterCopyWCCE.doOperation(OiifwClusterCopyWCCE.java:544)
at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
at oracle.sysman.oii.oiif.oiifw.OiifwActionsPhaseWCDE.doOperation(OiifwActionsPhaseWCDE.java:633)
at oracle.sysman.oii.oiif.oiifb.OiifbLinearIterator.iterate(OiifbLinearIterator.java:147)
at oracle.sysman.oii.oiic.OiicInstallAPISession$OiicAPISelCompsInstall.doOperation(OiicInstallAPISession.java:1058)
at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
at oracle.sysman.oii.oiic.OiicInstallAPISession.doInstallAction(OiicInstallAPISession.java:642)
at oracle.sysman.oii.oiic.OiicInstallAPISession.access$000(OiicInstallAPISession.java:88)
at oracle.sysman.oii.oiic.OiicInstallAPISession$OiicActionsThread.run(OiicInstallAPISession.java:934)
In the installActions.log file the last lines are:
INFO: ORACLE_HOME is not settable, hence not setting the value
INFO: {Parameter:TOPLEVEL_COMPONENT in {Aggregate:OuiConfigVariables:1.0.0.0.0:common}}: Parameter data type is not compatible with the provided String Array.
INFO: passing params to cf done
INFO: done saving info by cf
INFO: Updating files in Oracle home '/u01/app/11.2.0/grid' to remote nodes 'oradb2'.
INFO: InstallProgressMonitor: Starting phase 16
INFO: List of files to be excluded from:install/excludeFileList.txt
INFO: Updating files in Oracle home '/u01/app/11.2.0/grid' to remote nodes 'oradb2'.
INFO: Updating files in Oracle home '/u01/app/11.2.0/grid' to remote nodes 'oradb2'.
INFO: Running command '/tmp/OraInstall2009-12-14_01-12-56AM/mvstubs.sh' on the nodes 'oradb2'.
INFO: Invoking OUI on cluster nodes oradb2
INFO: /tmp/OraInstall2009-12-14_01-12-56AM/mvstubs.sh
INFO: Copying Oracle home '/u01/app/11.2.0/grid' to remote nodes 'oradb2'.
INFO: Copying Oracle home '/u01/app/11.2.0/grid' to remote nodes 'oradb2'.
Do you have any ideas? am trying hard to get it working...
Any ideas are welcomed... Thanks...

Answer 1

My 11gR2 Grid installation is stalled at 65% too, on "Performing remote operations".
I tried stopping the firewall on all of the remote (and the local) nodes, using the recommendation from above:
+"The workaround was that I stopped the firewall on both nodes using "*service iptables stop*" during installation - this seems to resolve the issue."+
but that didn't work for me. Although, my situation is a little different in that I'm not seeing the same "last line output" in installActions<Date>.log, I'm seeing:
INFO: setting ORACLE_HOME_NAME=Ora11g_gridinfrahome1. A default value was calculated as per oraparam.ini
INFO: Reading shiphome metadata from /mnt/oracle_media/grid/install/../stage/shiphomeproperties.xml
INFO: Loading beanstore from file:/mnt/oracle_media/grid/install/../stage/shiphomeproperties.xml
INFO: Translating external format into raw format
INFO: Restoring class oracle.install.driver.oui.ShiphomeMetadata from file:/mnt/oracle_media/grid/install/../stage/shiphomeproperties.xml
Which isn't very telling information concerning the problem I've encountered. I don't have anything in my oraInstall.err file.
This is a six node Oracle 11gR2 Grid that I'm trying to build on VMWare VMs running RHEL 5.4.
Here's the output of 'lsof -i' from the first node (dosfrdb01):
+[root@dosfrdb01 ~]# lsof -i+
COMMAND    PID  USER   FD   TYPE DEVICE SIZE NODE NAME
portmap   3294   rpc    3u  IPv4  11203       UDP *:sunrpc
portmap   3294   rpc    4u  IPv4  11204       TCP *:sunrpc (LISTEN)
mrouter   3780  root    8u  IPv4  11993       TCP *:8193 (LISTEN)
mrouter   3780  root    9u  IPv4  11994       TCP *:8194 (LISTEN)
mrouter   3780  root   12u  IPv4  12089       TCP *:8192 (LISTEN)
mrouter   3780  root   14u  IPv4 531742       TCP dosfrdb01-priv.oracle.dos:8194->dosfrdb01-priv.oracle.dos:50868 (ESTABLISHED)
magent    3786  root    9u  IPv4 531741       TCP dosfrdb01-priv.oracle.dos:50868->dosfrdb01-priv.oracle.dos:8194 (ESTABLISHED)
rpc.statd 3836  root    3u  IPv4  12121       UDP *:asf-rmcp
rpc.statd 3836  root    6u  IPv4  12112       UDP *:sco-websrvrmgr
rpc.statd 3836  root    7u  IPv4  12124       TCP *:asia (LISTEN)
sshd      3961  root    3u  IPv6 169837       TCP dosfrdb01.oracle.dos:ssh->dosdc01.2.168.192.in-addr.arpa:59590 (ESTABLISHED)
sshd      3963  grid    3u  IPv6 169837       TCP dosfrdb01.oracle.dos:ssh->dosdc01.2.168.192.in-addr.arpa:59590 (ESTABLISHED)
sshd      3963  grid    8u  IPv4 170091       TCP dosfrdb01.oracle.dos:6011 (LISTEN)
sshd      3963  grid    9u  IPv6 170092       TCP dosfrdb01.oracle.dos:6011 (LISTEN)
sshd      3963  grid   13u  IPv4 174216       TCP dosfrdb01.oracle.dos:6011->dosfrdb01.oracle.dos:43182 (ESTABLISHED)
java      4180  grid    8u  IPv4 174215       TCP dosfrdb01.oracle.dos:43182->dosfrdb01.oracle.dos:6011 (ESTABLISHED)
hpiod     4569  root    0u  IPv4  15595       TCP dosfrdb01.oracle.dos:2208 (LISTEN)
python    4574  root    4u  IPv4  15665       TCP dosfrdb01.oracle.dos:2207 (LISTEN)
sshd      4587  root    3u  IPv6  15742       TCP *:ssh (LISTEN)
cupsd     4596  root    4u  IPv4  15898       TCP dosfrdb01.oracle.dos:ipp (LISTEN)
cupsd     4596  root    6u  IPv4  15901       UDP *:ipp
ntpd      4626   ntp   16u  IPv4  16067       UDP *:ntp
ntpd      4626   ntp   17u  IPv6  16068       UDP *:ntp
+ntpd      4626   ntp   18u  IPv6  16069       UDP [fe80::250:56ff:fe91:33e3]:ntp+
ntpd      4626   ntp   19u  IPv6  16070       UDP dosfrdb01.oracle.dos:ntp
+ntpd      4626   ntp   20u  IPv6  16071       UDP [fe80::250:56ff:fe91:2dbc]:ntp+
ntpd      4626   ntp   21u  IPv4  16072       UDP dosfrdb01.oracle.dos:ntp
ntpd      4626   ntp   22u  IPv4  16073       UDP dosfrdb01-priv.oracle.dos:ntp
ntpd      4626   ntp   23u  IPv4  16076       UDP dosfrdb01.oracle.dos:ntp
sendmail  4644  root    4u  IPv4  16312       TCP dosfrdb01.oracle.dos:smtp (LISTEN)
avahi-dae 4748 avahi   13u  IPv4  17387       UDP *:mdns
avahi-dae 4748 avahi   14u  IPv6  17388       UDP *:mdns
avahi-dae 4748 avahi   15u  IPv4  17389       UDP *:47874
avahi-dae 4748 avahi   16u  IPv6  17390       UDP *:52836
gdm-binar 4828  root    3u  IPv4  18365       UDP *:xdmcp
In our case, as opposed to the post directly above, it doesn't seem like the other RAC nodes (e.g. dosfrdb02) have connected back to the first RAC node (dosfrdb01) using ssh or ractrans. There's a clue I suppose. However, I do see ssh connections:
sshd      3961  root    3u  IPv6 169837       TCP dosfrdb01.oracle.dos:ssh->dosdc01.2.168.192.in-addr.arpa:59590 (ESTABLISHED)
sshd      3963  grid    3u  IPv6 169837       TCP dosfrdb01.oracle.dos:ssh->dosdc01.2.168.192.in-addr.arpa:59590 (ESTABLISHED)
to the domain controller, dosdc01, where DHCP and DNS are running. Maybe the first RAC node is attempting to do a DNS lookup and the resolution for the VIPs in the grid's sub-domain isn't working. The VIP assignments should be delegated by DNS, to GNS, as I understand it, but I don't know if that's relevant during installation. I"ll have the SA check the DNS log file and configuration to see if he notices anything. I thought the network configuration was fine because the test that the installer performed to ensure that passwordless ssh ("user equivalence") works between RAC nodes executed successfully. I can also scp a file from one RAC node to another (as user 'grid') without providing a password.
We'll double check that the time is sync'ed up on all of the VMs too, but it ought to be.
The SA and I (the DBA) have carefully followed the instructions but obviously something isn't right. I didn't receive any unresolved errors during the installation, and now it's just stopped at 65%. Any ideas? Is there a script that I can run manually that would attempt to copy the files?
Thank you
Edited by: Bdub on Jun 18, 2010 10:10 AM
Edited by: Bdub on Jun 18, 2010 10:15 AM