Support for virtual IP in WKA list

Question

I am attempting to implement Coherence Grid Edition 3.5.3 Patch 7 . It's running on Sun Java 1.6.06 on a Solaris 5.10 machine (which I will call machine A1) which has an ip address 1.2.3.261. Also, there is a virtual IP address 1.2.3.260 which points to this same machine A1, as well as a hostname that maps to the virtual IP address of the machine A1.
(Note: I have mocked up the IP addresses in this post for security reasons).
I am using WKA due to concerns of the local network admins regarding multicast (debatable, but for now let's please assume I need to go with WKA).
If this production machine A1 develops any problem, we can quickly switch to an alternate machine A2. The alternate machine A2 contains automatically replicated copies of all of our important sub-directories (including the coherence xml configuration files), but it does have a different IP.
During a failure of machine A1, we would shut down machine A1, start up machine A2, and reroute the virtual IP to alternate machine A2's true IP. Machine A2 would be supplied a copy of the file-system that A1 had when A1 was shutdown.
We would seek to not have to edit any configuration files on A2. It is difficult to maintain alternate configuration files using our current replication scheme, and any machine-specific customizations are problematic.
Therefore, we would prefer to list the virtual IP of machines A1 and A2 in the WKA list, rather than the true IP of these machines.
However, when I attempt to list the virtual IP in the WKA list...
               <well-known-addresses>
                    <socket-address id="1">
                    <address>1.2.3.260</address> <!--virtual ip -->
                    <port>8088</port>
                    </socket-address>
               </well-known-addresses>
... I get the error below.
2010-09-03 12:01:51.464/1.401 Oracle Coherence GE 3.5.3/465p7 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2010-09-03 12:01:54.760/4.697 Oracle Coherence GE 3.5.3/465p7 <Error> (thread=Cluster, member=n/a): Node myhostname/1.2.3.261:8088 is not allowed to create a new cluster; WKA list: [1.2.3.260:8088]
2010-09-03 12:01:54.760/4.697 Oracle Coherence GE 3.5.3/465p7 <D5> (thread=Cluster, member=n/a): Service Cluster left the cluster
Exception in thread "main" java.lang.RuntimeException: Failed to start Service "Cluster" (ServiceState=SERVICE_STOPPED, STATE_ANNOUNCE)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:38)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:38)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:395)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:7)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:27)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:998)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:905)
at com.tangosol.net.DefaultCacheServer.start(DefaultCacheServer.java:139)
at com.tangosol.net.DefaultCacheServer.main(DefaultCacheServer.java:60)
2010-09-03 12:01:54.763/4.700 Oracle Coherence GE 3.5.3/465p7 <Error> (thread=main, member=n/a): Error while starting cluster: java.lang.RuntimeException: Failed to start Service "Cluster" (ServiceState=SERVICE_STOPPED, STATE_ANNOUNCE)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:38)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:38)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:395)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
What exactly is being checked here? Does each Coherence instance expect to see it's own IP listed in the WKA list, and refuse to run if it can't find it's own IP on the list?
Is the instance detecting it's own IP via an operating system call, or is there some way I can influence the instance's sense of what IP it is running on?
Is there any advice for supporting this kind of virtual IP scenario?
I have tried listing the hostname with no better success, because the hostname points to the virtual IP, and therefore has the same effect as listing the virtual IP.
As a workaround, I have altered the WKA list to list of the true IPs of both machines...
               <well-known-addresses>
                    <socket-address id="1">
                    <address>1.2.3.261</address> <!-- A1 -->
                    <port>8088</port>
                    </socket-address>
                    <socket-address id="2">
                    <address>1.2.3.262</address> <!-- A2 -->
                    <port>8088</port>
                    </socket-address>
               </well-known-addresses>
... even though both machines will never be up and running at the same time. This workaround avoids the error and the cluster appears to come up. I am not sure if there's any ill effect that at any given time one of these two IPs will never be up and running.
Despite my workaround, true support for virtual IPs would be superior; listing all the true non-virtual IPs in the WKA list means re-editing the list as the hardware changes.
Thanks for any advice!
P.S. I tried the above with Coherence 3.5.2 with the same results.
Edited by: user11114413 on Sep 3, 2010 11:45 AM

Answer 1

Hi user11114413,
The issue you are seeing actually has little to do with VIP, and more to do with there being multiple IP addresses for us to choose from on your box. For such multi-IP boxes, you'll want to tell us the IP to use, and in your case you want to tell as a VIP. This can be done either by editing your operational configuration file, and including an <address> element within the <unicast-listener> element, or via the tangosol.coherence.localhost system property. For example:
<unicast-listener>
    <well-known-addresses>
        <socket-address id="1">
            <address>1.2.3.260</address> <!--virtual ip -->
            <port>8088</port>
        </socket-address>
    </well-known-addresses>
    <address>1.2.3.260</address> <!--virtual ip -->
    <port>8088</port>
</unicast-listener>or
java ... -Dtangosol.coherence.localhost=1.2.3.260If you are using the same operational configuration on all nodes in your cluster then the system property approach is likely preferable, and would only be necessary on the two machines sharing the VIP.
As for using VIP or an extended WKA list, the choice is yours, either will work. If you do go the VIP route, it would obviously be a very bad idea to simultaneously use the same VIP and port at the same time from the two machines.
thanks,
Mark
Oracle Coherence