Unstable session replication in a HA cluster (CF10)

Question

Hi,
We have tried to create a HA cluster with requests being distributed round robin to N instances of coldfusion, we are NOT using sticky sessions as we are replication session state to all cf instances. What we are seing is that all is fine with low to moderate load, however under heavy load and at random times the replication fails and leads to things in session scope not working. This manifests in users not being able to login to our application (we store a token in session scope to store logged in status).
Again key point, under low to moderate load it all works fine, users are directed to random nodes in the cluster and their session is picked up fine as the session is distributed to all nodes,so pretty confident config is right.
Linux servers using CF10 with update 12 applied. Also running is fusion reactor 5.04 on all instances. Each instance has a 64GB heap, Java 7.0.15 (latest certified).
Firstly apache setup.
workers.properties
worker.list=balancer, jkstatus
worker.jkstatus.type=status
worker.balancer.type=lb
worker.balancer.balance_workers=cfusion_master,cfusion_slave2,cfusion_slave1
worker.balancer.method=R
worker.balancer.sticky_session=False
worker.balancer.ping_mode=A
worker.cfusion_master.type=ajp13
worker.cfusion_master.host=localhost
worker.cfusion_master.port=8012
worker.cfusion_master.max_reuse_connections=250
worker.cfusion_master.lbfactor=100
worker.cfusion_slave2.reference=worker.cfusion_master
worker.cfusion_slave2.port=8014
worker.cfusion_slave1.reference=worker.cfusion_master
worker.cfusion_slave1.port=8013
Now the server.xml from 2 nodes (as an example if I run a 2 node cluster)
One of the configs from a server in the cluster
<Server port="8007" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on">
  </Listener>
  <Listener className="org.apache.catalina.core.JasperListener">
  </Listener>
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener">
  </Listener>
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener">
  </Listener>
  <GlobalNamingResources>
    <Resource description="User database that can be updated and saved" name="UserDatabase" pathname="conf/tomcat-users.xml" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" type="org.apache.catalina.UserDatabase" auth="Container">
    </Resource>
  </GlobalNamingResources>
  <Service name="Catalina">
    <Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="150" namePrefix="catalina-exec-">
    </Executor>
    <Connector port="8012" protocol="AJP/1.3" connectionTimeout="600000" redirectPort="8445" tomcatAuthentication="false">
    </Connector>
    <Engine jvmRoute="cfusion" name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase">
        </Realm>
      </Realm>
      <Host name="localhost" autoDeploy="false" unpackWARs="true" appBase="webapps">
        <Valve pattern="%h %l %u %t &quot;%r&quot; %s %b" directory="logs" prefix="localhost_access_log." className="org.apache.catalina.valves.AccessLogValve" suffix=".txt" resolveHosts="false">
        </Valve>
      </Host>
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
        <Manager notifyListenersOnReplication="true" expireSessionsOnShutdown="false" className="org.apache.catalina.ha.session.DeltaManager">
        </Manager>
        <Channel className="org.apache.catalina.tribes.group.GroupChannel">
          <Membership port="45564" dropTime="3000" address="228.0.0.4" className="org.apache.catalina.tribes.membership.McastService" frequency="500">
          </Membership>
          <Receiver port="4001" autoBind="100" address="auto" selectorTimeout="5000" maxThreads="6" className="org.apache.catalina.tribes.transport.nio.NioReceiver">
          </Receiver>
          <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
            <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender">
            </Transport>
          </Sender>
          <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector">
          </Interceptor>
          <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor">
          </Interceptor>
        </Channel>
        <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="">
        </Valve>
        <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve">
        </Valve>
        <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener">
        </ClusterListener>
        <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener">
        </ClusterListener>
      </Cluster>
    </Engine>
    <Connector port="8499" protocol="org.apache.coyote.http11.Http11NioProtocol" connectionTimeout="20000" redirectPort="8443" executor="tomcatThreadPool">
    </Connector>
  </Service>
</Server>
Config from one of the other nodes
<Server port="8008" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on">
  </Listener>
  <Listener className="org.apache.catalina.core.JasperListener">
  </Listener>
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener">
  </Listener>
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener">
  </Listener>
  <GlobalNamingResources>
    <Resource description="User database that can be updated and saved" name="UserDatabase" pathname="conf/tomcat-users.xml" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" type="org.apache.catalina.UserDatabase" auth="Container">
    </Resource>
  </GlobalNamingResources>
  <Service name="Catalina">
    <Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="150" namePrefix="catalina-exec-">
    </Executor>
    <Connector port="8013" protocol="AJP/1.3" connectionTimeout="600000" redirectPort="8446" tomcatAuthentication="false">
    </Connector>
    <Engine jvmRoute="cfusion" name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase">
        </Realm>
      </Realm>
      <Host name="localhost" autoDeploy="false" unpackWARs="true" appBase="webapps">
        <Valve pattern="%h %l %u %t &quot;%r&quot; %s %b" directory="logs" prefix="localhost_access_log." className="org.apache.catalina.valves.AccessLogValve" suffix=".txt" resolveHosts="false">
        </Valve>
      </Host>
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
        <Manager notifyListenersOnReplication="true" expireSessionsOnShutdown="false" className="org.apache.catalina.ha.session.DeltaManager">
        </Manager>
        <Channel className="org.apache.catalina.tribes.group.GroupChannel">
          <Membership port="45564" dropTime="3000" address="228.0.0.4" className="org.apache.catalina.tribes.membership.McastService" frequency="500">
          </Membership>
          <Receiver port="4002" autoBind="100" address="auto" selectorTimeout="5000" maxThreads="6" className="org.apache.catalina.tribes.transport.nio.NioReceiver">
          </Receiver>
          <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
            <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender">
            </Transport>
          </Sender>
          <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector">
          </Interceptor>
          <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor">
          </Interceptor>
        </Channel>
        <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="">
        </Valve>
        <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve">
        </Valve>
        <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener">
        </ClusterListener>
        <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener">
        </ClusterListener>
      </Cluster>
    </Engine>
    <Connector port="8500" protocol="org.apache.coyote.http11.Http11NioProtocol" connectionTimeout="20000" redirectPort="8443" executor="tomcatThreadPool">
    </Connector>
  </Service>
</Server>
So what do i see in the logs?. Well sometimes I see exceptions like this
Mar 05, 2014 9:55:19 PM org.apache.catalina.ha.session.DeltaManager messageReceived
SEVERE: Manager [localhost#/]: Unable to receive message through TCP channel
java.lang.IllegalStateException: removeAttribute: Session already invalidated
          at org.apache.catalina.ha.session.DeltaSession.removeAttribute(DeltaSession.java:617)
          at org.apache.catalina.ha.session.DeltaRequest.execute(DeltaRequest.java:171)
          at org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1347)
          at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1293)
          at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:1014)
          at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListe ner.java:92)
          at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:897)
          at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:878)
          at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)
          at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:84)
          at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailu reDetector.java:113)
          at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:84)
          at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:84)
          at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.ja va:253)
          at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:2 87)
          at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTa sk.java:212)
          at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:1 01)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:722)
I'm unsure why this happens as tribes uses certified mesaging so it should have resent right?, in any case I believe I can change it so messages are not sent asynchronously, should sort this out.
I see (good) messages like this
Mar 05, 2014 9:42:19 PM org.apache.catalina.ha.session.DeltaManager startInternal
INFO: Register manager localhost#/ to cluster element Engine with name Catalina
Mar 05, 2014 9:42:19 PM org.apache.catalina.ha.session.DeltaManager startInternal
INFO: Starting clustering manager at localhost#/
Mar 05, 2014 9:42:19 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
INFO: Manager [localhost#/], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{192, 168, 128, 50}:4001,{192, 168, 128, 50},4001, alive=68824148, securePort=-1, UDP Port=-1, id={123 126 89 39 96 -59 69 8 -113 79 51 122 25 108 -11 -110 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds.
Mar 05, 2014 9:42:20 PM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions
INFO: Manager [localhost#/]; session state send at 3/5/14 9:42 PM received in 929 ms.
Mar 05, 2014 9:42:20 PM org.apache.catalina.ha.session.JvmRouteBinderValve startInternal
INFO: JvmRouteBinderValve started
So session state dies appear to be flying around the cluster, I do nightly restarts of some of the nodes due to another issue I have with an ever growing heap (separate issue), interestingly I also see nodes leave and join the cluster, again this is good (shows the multicast is working, and also that replication should be working).
Mar 05, 2014 2:30:16 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{192, 168, 128, 50}:4001,{192, 168, 128, 50},4001, alive=18629101, securePort=-1, UDP Port=-1, id={-2 65 10 -79 53 -75 76 52 -99 63 -90 -120 34 -89 -14 100 }, payload={}, command={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
Mar 05, 2014 2:30:16 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberDisappeared
INFO: Received member disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{192, 168, 128, 50}:4001,{192, 168, 128, 50},4001, alive=18629101, securePort=-1, UDP Port=-1, id={-2 65 10 -79 53 -75 76 52 -99 63 -90 -120 34 -89 -14 100 }, payload={}, command={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]
Mar 05, 2014 2:35:16 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{192, 168, 128, 50}:4001,{192, 168, 128, 50},4001, alive=1083, securePort=-1, UDP Port=-1, id={123 126 89 39 96 -59 69 8 -113 79 51 122 25 108 -11 -110 }, payload={}, command={}, domain={}, ]
So stuck now on how to proceed, to establish why at random times the replication fails, leading to cluster collapse. Could it be the size of the session?, I have a few CFCs stuffed into session scope, but perhaps when the load is high there is too many?. Things fail even with a cluster of 2 on one server, initially I had a 8 node cluster on 2 separate machines but when it failed it rolled it back to a cluster of 2 instances on the one server to see if that was stable (its not 100% which is what I need).
Any advice, points gratefully received.

Answer 1

So Lynux, that’s an interesting sounding solution. Would be great if it made some difference for Guitsboy. We’ll see. I notice that you say you’ve not yet tried it, though, and fair enough. Thanks for offering it.
But I’m curious: did you ever resolve your original problem? And if not, hopefully you saw the note I just wrote to Guitsboy, asking him something that may well interest you if you still have your problem. On rereading this thread, from back in April, I’ve also had some new thoughts come to mind which I’ll share, if it may help either of you, or others with this seeming same issue.
To remind readers who may not want to review the whole thread, you had said originally that “all is fine with low to moderate load, however under heavy load and at random times the replication fails“, and that this failure “manifests in users not being able to login to our application (we store a token in session scope to store logged in status)”.  Then it seems you may have concluded that things were down to the error you were seeing in the logs:
Mar 05, 2014 9:55:19 PM org.apache.catalina.ha.session.DeltaManager messageReceived
SEVERE: Manager : Unable to receive message through TCP channel
java.lang.IllegalStateException: removeAttribute: Session already invalidated
And now guitsboy reports seeing the same error.
But here’s the thing that came to mind for me tonight as I read this: you know, there can be a lot of other reasons that users can feel that they “lose their session”, even without using clustering and replication.
There are issues related sometimes to folks having duplicate session tokens (which can happen for various reasons, including perhaps ones in your code, and maybe only when people visit pages in a certain pattern, so that it happens only occasionally and not always).
Then there is an issue that can arise if you are supporting both http and https requests, where Tomcat (not CF) balks at that (see http://www.petefreitag.com/item/817.cfm, and though he shows a solution in IIS you should be able to implement a similar one in mod_rewrite if that was indeed perhaps your issue).
So I’d be curious if either of you may be in a position to have a failing client use any sort of client tool (like Chrome’s dev tools, or Firebug or Firefox’s new builtin tools, or IE’s f12 dev tools) to watch the communication between the client and the server, and especially to watch the cookies being sent. You guys both mention using jsessionid. Are they the same cookie value on each request? And/or are there more than jsessionid? I’ve seen it happen. There could be differences in the domain property reported for the cookie, the httponly property, the secure property, and so on.  And you really do want to view the value sent from the client to the server, because if you view the cookie scope on the server a) it may show values set ON the server rather than sent TO the server, and b) it won’t show these additional cookie properties that were in play on the client. CF only sees the cookie name and value.
I’ve helped many people find out that this was the reason for the seeming session loss (and sometimes it was not all requests by all clients but perhaps only some requests for some clients, all on the same server). At least if this is the crux of the problem, you can then tackle WHY it’s happening.  There can be many reasons, from code to configuration, so I won’t belabor them now.
But if either of you may be able to confirm this, perhaps we can help you both get a little closer to a real explanation and solution for your problem. Again, I’m just guessing a bit based on what you’ve written. I realize it may be that none of this is the problem and you have hit some other real unrelated bug. But I really feel confident that you ought to try to check this out first, as it’s indeed been the crux of problems for others, without respect to clustering.  It seems worth ruling out, so that you don’t get misled chasing the problem on the assumption that it is about clustering.
As always, hope that helps.
/charlie