Unable to see other OC4J nodes in a cluster

Question

I have installed 2 instances of OracleAS on 2 separate machines, both machines ( Lnx-5 and Lnx-6 ) were installed with the J2EE component and WEB component.
During installation, I have selected Lnx-5 as the administration node of the cluster, and I have configured the discovery address using multicast address 225.0.0.33:8001.
There were no installations errors encountered and things seems to work fine.
However, on Lnx-5, it can't "see" Lnx-6 as one of its cluster nodes. On both Lnx-5 and Lnx-6, I see the following when I issued the "opmnctl @cluster status".
---- On Lnx-5 , here is what I got ---------
[root@Lnx-5 conf]# opmnctl @cluster status
Processes in Instance: Lnx5.anydomain.com
--------------------------------------------------------------+---------
ias-component | process-type | pid | status
--------------------------------------------------------------+---------
OC4JGroup:default_group | OC4J:home | 5392 | Alive
ASG | ASG | N/A | Down
HTTP_Server | HTTP_Server | 5391 | Alive
---- On Lnx-6 , here is what I got ---------
[root@Lnx-6 conf]# opmnctl @cluster status
Processes in Instance: Lnx6.anydomain.com
--------------------------------------------------------------+---------
ias-component | process-type | pid | status
--------------------------------------------------------------+---------
OC4JGroup:default_group | OC4J:home | 5392 | Alive
ASG | ASG | N/A | Down
HTTP_Server | HTTP_Server | 5391 | Alive
I suppose I should see both Lnx-5 and Lnx-6 when I issue the commad in either nodes.
I have also verified that both machine are synchronized to the NTP server.
I have also done a tcpdump on both nodes, indeed I can multicast ( 225.0.0.33:8001 ) packets arriving at both nodes..
Really need some help in what would have go wrong, what information should I look for to address this issue.
Thanks in advance!!

Answer 1

Ok, for the discovery server configuration, here is the config that I have in the opmn.xml file, both lnx-5 and lnx-6 use exactly the same configuration :
<notification-server interface="ipv4">
<port local="6101" remote="6201" request="6004"/>
<ssl enabled="true" wallet-file="$ORACLE_HOME/opmn/conf/ssl.wlt/default"/>
<topology>
          <discover list="10.1.230.11:6201,10.1.230.12:6201"/>
</topology>
</notification-server>
the ip address of Lnx-5 is 10.1.230.11, and Lnx-6 is 10.1.230.12.
Once this was configured on both Lnx-5, Lnx-6, I keep seeing this error from the Lnx-6's log file :
07/05/16 22:10:18 [pm-process] Process Alive: default_group~home~default_group~1
(1542677438:3859)
07/05/16 22:10:18 [pm-requests] Request 2 Completed. Command: /start
07/05/16 22:13:25 [ons-connect] Connection 9,10.1.230.11,6201 connect (Connectio
n refused)
07/05/16 22:13:26 [ons-connect] Connection a,10.1.230.12,6201 connect (Connectio
n refused)
Well, Once I enabled the debugging, there were some errors reported when opmn is started, the errors are as follows :
Loading Module libopmnohs callback functions
Module libopmnohs: loaded callback function opmnModInitialize
Module libopmnohs: unable to load callback function opmnModSetNumProcs
Module libopmnohs: unable to load callback function opmnModParse
Module libopmnohs: unable to load callback function opmnModDebug
Module libopmnohs: unable to load callback function opmnModDepend
Module libopmnohs: loaded callback function opmnModStart
Module libopmnohs: unable to load callback function opmnModReady
Module libopmnohs: loaded callback function opmnModNotify
Module libopmnohs: loaded callback function opmnModRestart
Module libopmnohs: loaded callback function opmnModStop
Module libopmnohs: loaded callback function opmnModPing
Module libopmnohs: loaded callback function opmnModProcRestore
Module libopmnohs: loaded callback function opmnModProcComp
Module libopmnohs: unable to load callback function opmnModReqComp
Module libopmnohs: unable to load callback function opmnModCall
Module libopmnohs: unable to load callback function opmnModInfo
Module libopmnohs: unable to load callback function opmnModCron
Module libopmnohs: loaded callback function opmnModTerminate
Loading Module libopmnoc4j callback functions
Module libopmnoc4j: loaded callback function opmnModInitialize
Module libopmnoc4j: unable to load callback function opmnModSetNumProcs
Module libopmnoc4j: loaded callback function opmnModParse
Module libopmnoc4j: unable to load callback function opmnModDebug
Module libopmnoc4j: unable to load callback function opmnModDepend
Module libopmnoc4j: loaded callback function opmnModStart
Module libopmnoc4j: unable to load callback function opmnModReady
Module libopmnoc4j: loaded callback function opmnModNotify
Module libopmnoc4j: loaded callback function opmnModRestart
Module libopmnoc4j: loaded callback function opmnModStop
Module libopmnoc4j: loaded callback function opmnModPing
Module libopmnoc4j: loaded callback function opmnModProcRestore
Module libopmnoc4j: loaded callback function opmnModProcComp
Module libopmnoc4j: unable to load callback function opmnModReqComp
Module libopmnoc4j: unable to load callback function opmnModCall
Module libopmnoc4j: unable to load callback function opmnModInfo
Module libopmnoc4j: unable to load callback function opmnModCron
Module libopmnoc4j: loaded callback function opmnModTerminate
Loading Module libopmncustom callback functions
Module libopmncustom: loaded callback function opmnModInitialize
Module libopmncustom: unable to load callback function opmnModSetNumProcs
Module libopmncustom: loaded callback function opmnModParse
Module libopmncustom: loaded callback function opmnModDebug
Module libopmncustom: unable to load callback function opmnModDepend
Module libopmncustom: loaded callback function opmnModStart
Module libopmncustom: loaded callback function opmnModReady
Module libopmncustom: unable to load callback function opmnModNotify
Module libopmncustom: loaded callback function opmnModRestart
Module libopmncustom: loaded callback function opmnModStop
Module libopmncustom: loaded callback function opmnModPing
Module libopmncustom: loaded callback function opmnModProcRestore
Module libopmncustom: loaded callback function opmnModProcComp
Module libopmncustom: loaded callback function opmnModReqComp
Module libopmncustom: unable to load callback function opmnModCall
Module libopmncustom: unable to load callback function opmnModInfo
Module libopmncustom: unable to load callback function opmnModCron
Module libopmncustom: loaded callback function opmnModTerminate
Loading Module libopmniaspt callback functions
Module libopmniaspt: loaded callback function opmnModInitialize
Module libopmniaspt: unable to load callback function opmnModSetNumProcs
Module libopmniaspt: unable to load callback function opmnModParse
Module libopmniaspt: unable to load callback function opmnModDebug
Module libopmniaspt: unable to load callback function opmnModDepend
Module libopmniaspt: loaded callback function opmnModStart
Module libopmniaspt: loaded callback function opmnModReady
Module libopmniaspt: unable to load callback function opmnModNotify
Module libopmniaspt: unable to load callback function opmnModRestart
Module libopmniaspt: loaded callback function opmnModStop
Module libopmniaspt: unable to load callback function opmnModPing
Module libopmniaspt: unable to load callback function opmnModProcRestore
Module libopmniaspt: loaded callback function opmnModProcComp
Module libopmniaspt: unable to load callback function opmnModReqComp
Module libopmniaspt: unable to load callback function opmnModCall
Module libopmniaspt: unable to load callback function opmnModInfo
Module libopmniaspt: unable to load callback function opmnModCron
Module libopmniaspt: loaded callback function opmnModTerminate
Looks pretty bad.. What cuases those errors to happen? Are they related?
Thanks!!