Getting error: java.lang.IllegalStateException: Service has been started by a different configurable cache factory.

Question

We are trying to implement host name based security for our Coherence cluster. This is mainly to prevent processes running in one environment (UAT for example) connecting to the Coherence cluster running in another environment (PROD for example).
We wanted to ensure that 1) Only Coherence nodes running on specific machines can join the cluster as members. To accomplish this, we added the list of authorized hosts to the override file (see marketrisk-coherence-override.xml) This works as expected with no issues
We also want to ensure that we allow client connections from some specific machines. All of our clients connect to the cluster through the proxy. So, we added added a host filter to the cache config file (see marketrisk-cache-config.xml) to control the list of client machines that can are allowed to connect to the cluster. The host filter is pretty simple. If the incoming host address is found in a cache, it allows the connection else it refuses the connection. Please see AuthorizedClientHostsFilter.java for more details.
When we start the cluster, the storage nodes are failing to come up with the following error message:
Exception in thread "main" java.lang.IllegalStateException: Service "ReferenceDataCacheService" has been started by a different configurable cache factory.
                at com.tangosol.net.ExtensibleConfigurableCacheFactory.validateBackingMapManager(ExtensibleConfigurableCacheFactory.java:829)
                at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureService(ExtensibleConfigurableCacheFactory.java:585)
                at com.tangosol.net.ExtensibleConfigurableCacheFactory.startServices(ExtensibleConfigurableCacheFactory.java:666)
                at com.tangosol.net.DefaultCacheServer.startServicesInternal(DefaultCacheServer.java:369)
                at com.tangosol.net.DefaultCacheServer.initialStartServices(DefaultCacheServer.java:466)
                at com.tangosol.net.DefaultCacheServer.startAndMonitor(DefaultCacheServer.java:74)
                at com.tangosol.net.DefaultCacheServer.main(DefaultCacheServer.java:260)
This error goes away and the cluster starts normally if we remove host filter from the cache config file or we modify AuthorizedClientHostsFilter.java to not get a handle to the cache when it is instantiated. Please see AuthorizedClientHostsFilter1.java which has this change.
We believe this is happening because the storage node is creating two cache factories when we use the host filter (AuthorizedClientHostsFilter.java). The storage nodes creates only one cache factory if  we do not use the host filter or if I use AuthorizedClientHostsFilter1.java.
I do not understand why this is happening. Can you please help? The attached zip file should have all the information that you will probably need.

Answer 1

public class AuthorizedClientHostsFilter implements Filter {
    //private static final Logger logger = LoggerFactory.getLogger(AuthorizedClientHostsFilter.class);
    public static final String ALL_HOSTS = "ALL_HOSTS";
    private static NamedCache hostsCache = new ContinuousQueryCache(
            CacheFactory.getCache("client-auth-hosts"), AlwaysFilter.INSTANCE, false);
    public AuthorizedClientHostsFilter() {
    public boolean evaluate(Object o) {
        logger.info("hostsCache.size() = "+hostsCache.size());
        Set<String> authorizedHostsinCache = hostsCache.keySet();
        for (String hostName : authorizedHostsinCache) {
            logger.info("Host: "+hostName);
        if (hostsCache.containsKey("ALL_HOSTS")) return true;
        InetAddress address = (InetAddress) o;
        //logger.info("Incoming Host: "+address.getHostAddress());
        return hostsCache.containsKey(address.getHostAddress());
public class AuthorizedClientHostsFilter1 implements Filter {
    //private static final Logger logger = LoggerFactory.getLogger(AuthorizedClientHostsFilter.class);
    public static final String ALL_HOSTS = "ALL_HOSTS";
    private static NamedCache hostsCache = null;
    public AuthorizedClientHostsFilter1() {
    public boolean evaluate(Object o) {
        if (hostsCache == null) {
            hostsCache = new ContinuousQueryCache(
                    CacheFactory.getCache("client-auth-hosts"), AlwaysFilter.INSTANCE, false);
        logger.info("hostsCache.size() = "+hostsCache.size());
        Set<String> authorizedHostsinCache = hostsCache.keySet();
        for (String hostName : authorizedHostsinCache) {
            logger.info("Host: "+hostName);
        if (hostsCache.containsKey("ALL_HOSTS")) return true;
        InetAddress address = (InetAddress) o;
        //logger.info("Incoming Host: "+address.getHostAddress());
        return hostsCache.containsKey(address.getHostAddress());