Extend-TCP client not failing over to another proxy after machine failure

Question

I have a configuration of three hosts. on A is the client, on B & C are a proxy and a cache instance. I've defined an AddressProvider that returns the address of B and then C. The client just repeatedly calls the cache (read-only). The client configuration is:
<?xml version="1.0"?>
<cache-config
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config
coherence-cache-config.xsd">
<caching-scheme-mapping>
<cache-mapping>
<cache-name>cache1</cache-name>
<scheme-name>extend-near</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<!-- Use ExtendTCP to connect to a proxy. -->
<caching-schemes>
<near-scheme>
<scheme-name>extend-near</scheme-name>
<front-scheme>
<local-scheme>
<high-units>1000</high-units>
</local-scheme>
</front-scheme>
<back-scheme>
<remote-cache-scheme>
<scheme-ref>remote-cache1</scheme-ref>
</remote-cache-scheme>
</back-scheme>
<invalidation-strategy>all</invalidation-strategy>
</near-scheme>
<remote-cache-scheme>
<scheme-name>remote-cache1</scheme-name>
<service-name>cache1ExtendedTcpProxyService</service-name>
<initiator-config>
<tcp-initiator>
<remote-addresses>
<address-provider>
<class-name>com.foo.clients.Cache1AddressProvider</class-name>
</address-provider>
</remote-addresses>
<connect-timeout>10s</connect-timeout>
</tcp-initiator>
<outgoing-message-handler>
<request-timeout>5s</request-timeout>
</outgoing-message-handler>
</initiator-config>
</remote-cache-scheme>
</caching-schemes>
If I shutdown the proxy that the client is connected to on host B, failover occurs quickly by calling the AddressProvider. But if I shut down the network for host B (or drop the TCP port of the proxy on B) to simulate a machine failure, failover does not occur. The client simply continues to try to contact B and dutifully times out in 5 seconds. It never asks the AddressProvider for another address.
How do I get failover to kick in?

Answer 1

Hello,
If you are testing Coherence*Extend failover in the face of a network, machine, or NIC failure, you should enable Connection heartbeats on both the <tcp-initiator/> and <tcp-acceptor/>. For example:
Client cache config:
<remote-cache-scheme>
  <scheme-name>extend-direct</scheme-name>
  <service-name>ExtendTcpCacheService</service-name>
  <initiator-config>
    <tcp-initiator>
      <remote-addresses>
        <socket-address>
          <address system-property="tangosol.coherence.extend.address">localhost</address>
          <port system-property="tangosol.coherence.extend.port">9099</port>
        </socket-address>
      </remote-addresses>
      <connect-timeout>2s</connect-timeout>
    </tcp-initiator>
    <outgoing-message-handler>
      <heartbeat-interval>10s</heartbeat-interval>
      <heartbeat-timeout>5s</heartbeat-timeout>
      <request-timeout>15s</request-timeout>
    </outgoing-message-handler>
  </initiator-config>
</remote-cache-scheme>Proxy cache config:
<proxy-scheme>
  <scheme-name>example-proxy</scheme-name>
  <service-name>ExtendTcpProxyService</service-name>
  <thread-count system-property="tangosol.coherence.extend.threads">2</thread-count>
  <acceptor-config>
    <tcp-acceptor>
      <local-address>
        <address system-property="tangosol.coherence.extend.address">localhost</address>
        <port system-property="tangosol.coherence.extend.port">9099</port>
      </local-address>
    </tcp-acceptor>
    <outgoing-message-handler>
      <heartbeat-interval>10s</heartbeat-interval>
      <heartbeat-timeout>5s</heartbeat-timeout>
      <request-timeout>15s</request-timeout>
    </outgoing-message-handler>
  </acceptor-config>
  <autostart system-property="tangosol.coherence.extend.enabled">true</autostart>
</proxy-scheme>This is because it may take the TCP/IP stack a considerable amount of time to detect that it's peer is unavailable after a network, machine, or NIC failure (O/S dependent).
Jason