EM for Coherence - Cannot automatically start departed storage enabled nodes

Question

Hi Guru,
I have a cluster with 4 storage enabled nodes. I want EM to monitor those 4 storage enabled nodes and automatically bring up nodes if  down.  So i set the "Nodes Replenish and Entity Discovery Alert Metric ->
Cluster Size Change (To Replenish Nodes)" as follows:
         Warning Threshold: Not Defined
          Critical Threshold: 3
          Comparision Operator: <
          Occurrences Before Alert: 1
I manually killed 2 storage nodes and hope EM can automatically bring up them. But unfortunately, this never happen. I even cannot see the correct "Severity Message" displayed on GUI, it always shows "0 nodes departed Coherence cluster". 
Did anyone have the similar problem? Any hints are appreciated!
Thanks
Hysun

Answer 1

Hi,
It looks like your cache servers have not used the correct cache configuration file so they do not have a service with the name DistributedSessions. You can see this in your log output here:
Services
  ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=5}
  InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=5}
  PartitionedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
  ReplicatedCache{Name=ReplicatedCache, State=(SERVICE_STARTED), Id=3, Version=3.0, OldestMemberId=2}
  Optimistic{Name=OptimisticCache, State=(SERVICE_STARTED), Id=4, Version=3.0, OldestMemberId=2}
  InvocationService{Name=InvocationService, State=(SERVICE_STARTED), Id=5, Version=3.1, OldestMemberId=2}
You said in the original post that you used the following JVM arguments:
-Dtangosol.coherence.distributed.localstorage=true -Dtangosol.coherence.session.localstorage=true -Dtangosol.coherence.cluster=CoherenceCluster -Dtangosol.coherence.clusteraddress=231.1.3.4 -Dtangosol.coherence.clusterport=7744
...but none of those specify the cache configuration to use (in your case session-cache-config.xml) so the node will use the default configuration file; the default name is coherence-cache-config.xml which is inside the Coherence jar file.
You need to add the following JVM argument
-Dtangosol.coherence.cacheconfig=session-cache-config.xml -Dtangosol.coherence.distributed.localstorage=true -Dtangosol.coherence.session.localstorage=true -Dtangosol.coherence.cluster=CoherenceCluster -Dtangosol.coherence.clusteraddress=231.1.3.4 -Dtangosol.coherence.clusterport=7744
JK