Urgent! Node keep disconnecting from Coherence Cluster

The system consists of 4 standalone cache servers with local storage set to true and 14 other embedded nodes started with different web apps on tomcat with local storage set to false.
When the servers are started after a new deployment, sometimes it would just work, but most times some random tomcat server will stuck in the following pattern.
First it would successful start the cluster service and join an existing cluster.
Oracle Coherence Version 3.5.1/461
Grid Edition: Development mode
Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
2012-07-18 12:24:33.335/31.845 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2012-07-18 12:24:33.550/32.060 Oracle Coherence GE 3.5.1/461 <Info> (thread=Cluster, member=n/a): This Member(Id=8, Timestamp=2012-07-18 12:24:33.347, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:20831, Role=ApacheCatalinaStartupBootstrap, Edition=Grid Edition, Mode=Development, CpuCount=24, SocketCount=24) joined cluster "DEV1" with senior Member(Id=10, Timestamp=2012-07-18 09:39:44.861, Address=10.34.32.101:8090, MachineId=2149, Location=machine:dev1ssapp3,process:27796, Role=ApacheCatalinaStartupBootstrap, Edition=Grid Edition, Mode=Development, CpuCount=64, SocketCount=64)
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=1, Timestamp=2012-07-18 12:22:14.231, Address=10.34.32.107:8090, MachineId=2155, Location=machine:dev1sxapp2,process:1278, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=2, Timestamp=2012-07-18 12:22:14.331, Address=10.34.32.106:8089, MachineId=2154, Location=machine:dev1sxapp1,process:6549, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=3, Timestamp=2012-07-18 12:22:55.086, Address=10.34.32.106:8088, MachineId=2154, Location=machine:dev1sxapp1,process:23083, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=4, Timestamp=2012-07-18 12:22:56.799, Address=10.34.32.107:8088, MachineId=2155, Location=machine:dev1sxapp2,process:19624, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=5, Timestamp=2012-07-18 12:24:31.869, Address=10.34.32.106:8090, MachineId=2154, Location=machine:dev1sxapp1,process:24411, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=6, Timestamp=2012-07-18 12:24:33.084, Address=10.34.32.101:8088, MachineId=2149, Location=machine:dev1ssapp3,process:28932, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=14, Timestamp=2012-07-18 09:40:50.645, Address=10.34.32.104:8090, MachineId=2152, Location=machine:dev1ssapp4,process:17697, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.556/32.066 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=17, Timestamp=2012-07-18 10:35:16.722, Address=10.34.32.104:8093, MachineId=2152, Location=machine:dev1ssapp4,process:19365, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.556/32.066 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=18, Timestamp=2012-07-18 10:38:47.714, Address=10.34.32.101:8093, MachineId=2149, Location=machine:dev1ssapp3,process:29887, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.563/32.073 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 10 joined Service Management with senior member 10
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 10
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 2 joined Service Management with senior member 10
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 4 joined Service Management with senior member 10
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 17 joined Service Management with senior member 10
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 17 joined Service PFExpiryDistributedCache with senior member 17
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 17 joined Service SsoRuleEntryDistributedCache with senior member 17
2012-07-18 12:24:33.567/32.077 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 3 joined Service Management with senior member 10
2012-07-18 12:24:33.567/32.077 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 18 joined Service Management with senior member 10
2012-07-18 12:24:33.567/32.077 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 18 joined Service PFExpiryDistributedCache with senior member 17
2012-07-18 12:24:33.567/32.077 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 18 joined Service SsoRuleEntryDistributedCache with senior member 17
2012-07-18 12:24:33.568/32.078 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 14 joined Service Management with senior member 10
2012-07-18 12:24:33.568/32.078 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 5 joined Service Management with senior member 10
2012-07-18 12:24:33.579/32.089 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 6 joined Service Management with senior member 10
Then it started getting heartbeat overdue message and cluster stopped:
2012-07-18 12:37:20.717/799.227 Oracle Coherence GE 3.5.1/461 <Info> (thread=PacketListenerN, member=8): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2012-07-18 12:37:29.916/808.426 Oracle Coherence GE 3.5.1/461 <Info> (thread=PacketListenerN, member=8): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2012-07-18 12:37:59.291/837.801 Oracle Coherence GE 3.5.1/461 <Error> (thread=PacketListenerN, member=8): Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: Unable to refresh sockets: [UnicastUdpSocket{State=STATE_OPEN, address:port=10.34.32.107:8089}, MulticastUdpSocket{State=STATE_OPEN, address:port=237.0.0.1:40109, InterfaceAddress=10.34.32.107, TimeToLive=4}, TcpSocketAccepter{State=STATE_OPEN, ServerSocket=10.34.32.107:8089}]; last failed socket: MulticastUdpSocket{State=STATE_OPEN, address:port=237.0.0.1:40109, InterfaceAddress=10.34.32.107, TimeToLive=4}
at com.tangosol.coherence.component.net.Cluster$SocketManager.refreshSockets(Cluster.CDB:91)
at com.tangosol.coherence.component.net.Cluster$SocketManager$MulticastUdpSocket.onInterruptedIOException(Cluster.CDB:9)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:33)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: Receive timed out
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
at java.net.DatagramSocket.receive(DatagramSocket.java:725)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
2012-07-18 12:37:59.291/837.801 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=8): Service Cluster left the cluster
2012-07-18 12:37:59.293/837.803 Oracle Coherence GE 3.5.1/461 <D5> (thread=Invocation:Management, member=8): Service Management left the cluster
2012-07-18 12:37:59.293/837.803 Oracle Coherence GE 3.5.1/461 <D5> (thread=ReplicatedCache:HibernateReplicatedCache, member=8): Service HibernateReplicatedCache left the cluster
Then it started getting messages from various nodes about the existing cluster:
2012-07-18 12:40:02.862/961.372 Oracle Coherence GE 3.5.1/461 <Info> (thread=queue://authenticationService.logonEvent.consumer-2, member=n/a): Restarting cluster
2012-07-18 12:40:02.891/961.401 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2012-07-18 12:40:20.167/978.677 Oracle Coherence GE 3.5.1/461 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2012-07-18 12:40:02.867, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:20831, Role=ApacheCatalinaStartupBootstrap) has been attempting to join the cluster at address 237.0.0.1:40109 with TTL 4 for 17 seconds without success; this could indicate a mis-configured TTL value, or it may simply be the result of a busy cluster or active failover.
2012-07-18 12:40:20.168/978.678 Oracle Coherence GE 3.5.1/461 <Warning> (thread=Cluster, member=n/a): Received a discovery message that indicates the presence of an existing cluster:
Message "NewMemberAnnounceWait"
FromMember=Member(Id=4, Timestamp=2012-07-18 12:22:56.799, Address=10.34.32.107:8088, MachineId=2155, Location=machine:dev1sxapp2,process:19624, Role=ApacheCatalinaStartupBootstrap)
FromMessageId=0
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=9
ToPollId=0
Poll=null
Packets
[000]=Broadcast{PacketType=0x0DDF00D2, ToId=0, FromId=4, Direction=Incoming, ReceivedMillis=12:40:20.167, MessageType=9, MessagePartCount=1, MessagePartIndex=0, Body=0x00000001389AE63E1F0A22206B00000000000000000000000040001
F980000086B000405011818044445563140400A64657631737861707032053139363234401E417061636865436174616C696E6153746172747570426F6F7473747261700001000001389AF5E6330A22206B00000000000000000000000040001F990000086B000005011818044445563140
400A64657631737861707032053230383331401E417061636865436174616C696E6153746172747570426F6F74737472617000000001389AE6376E0A22206A00000000000000000000000040001F980000086A000305011818044445563140400A646576317378617070310532333038334
01E417061636865436174616C696E6153746172747570426F6F74737472617000, Body.length=287}
Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_ANNOUNCE), Id=0, Version=3.5}
ToMemberSet=null
NotifySent=false
ToMember=Member(Id=0, Timestamp=2012-07-18 12:40:02.867, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:20831, Role=ApacheCatalinaStartupBootstrap)
SeniorMember=Member(Id=3, Timestamp=2012-07-18 12:22:55.086, Address=10.34.32.106:8088, MachineId=2154, Location=machine:dev1sxapp1,process:23083, Role=ApacheCatalinaStartupBootstrap)
Then it failed to connect to the cluster:
2012-07-18 12:40:33.187/991.697 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster left the cluster
2012-07-18 12:40:33.190/991.700 Oracle Coherence GE 3.5.1/461 <Error> (thread=queue://authenticationService.logonEvent.consumer-2, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException:
Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=ServiceMemberSet(
OldestMember=n/a
ActualMemberSet=MemberSet(Size=0, BitSetCount=0
MemberId/ServiceVersion/ServiceJoined/ServiceLeaving
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:38)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:366)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:7)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:27)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:1011)
at com.tangosol.coherence.hibernate.CoherenceCacheProvider.nextTimestamp(CoherenceCacheProvider.java:58)
at org.hibernate.cache.impl.bridge.RegionFactoryCacheProviderBridge.nextTimestamp(RegionFactoryCacheProviderBridge.java:93)
at org.hibernate.impl.SessionFactoryImpl.openSession(SessionFactoryImpl.java:652)
at org.hibernate.ejb.EntityManagerImpl.getRawSession(EntityManagerImpl.java:111)
at org.hibernate.ejb.EntityManagerImpl.getSession(EntityManagerImpl.java:91)
at org.hibernate.ejb.AbstractEntityManagerImpl.setDefaultProperties(AbstractEntityManagerImpl.java:250)
at org.hibernate.ejb.AbstractEntityManagerImpl.postInit(AbstractEntityManagerImpl.java:162)
at org.hibernate.ejb.EntityManagerImpl.<init>(EntityManagerImpl.java:84)
at org.hibernate.ejb.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:112)
at org.hibernate.ejb.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:107)
at org.springframework.orm.jpa.JpaTransactionManager.createEntityManagerForTransaction(JpaTransactionManager.java:399)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:321)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:335)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
at java.lang.Thread.run(Thread.java:662)
2012-07-18 12:40:33.216/991.726 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2012-07-18 12:40:50.398/1008.908 Oracle Coherence GE 3.5.1/461 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2012-07-18 12:40:33.194, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:20831, Role=ApacheCatalinaStartupBootstrap) has been attempting to join the cluster at address 237.0.0.1:40109 with TTL 4 for 17 seconds without success; this could indicate a mis-configured TTL value, or it may simply be the result of a busy cluster or active failover.
This particular jvm then start go into this kind of loop, receive a lot of messages from other nodes about the exist cluster but failed to join.
I have ran the MulticastTest and Datagram test which didn't reveal any obvious network issue. What should I do next?
JVM is 1.6.0_31
Thanks a lot in advance, any help will be greatly appreciated.

I correlated the log with all servers and found the issue might be due to some member it is connected with actually was being restarted.
Server 1:
- starts as member 23 and discovered the existing cluster and joined it. Then a lot of messages on server1 with all different members joining the cluster with different member id.
- Then it found some member failed to respond:
2012-07-30 22:00:25.371/34.325 Oracle Coherence GE 3.5.1/461 <D6> (thread=PacketPublisher, member=n/a): Member(Id=5, Timestamp=2012-07-30 15:35:09.735, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:21324, Role=ApacheCatalinaStartupBootstrap) has failed to respond to 17 packets; declaring this member as paused.
- Then it's requesting the departure confirmation for member 5:
2012-07-30 22:00:52.042/60.996 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=n/a): Timeout while delivering a packet Directed{PacketType=0x0DDF00D5, ToId=0, FromId=23, Direction=Outgoing, SentCount=145, SentMillis=22:00:51.832, ToMemberSet=[5(1)], ServiceId=0, MessageType=16, FromMessageId=6, ToMessageId=0, MessagePartCount=1, MessagePartIndex=0, NackInProgress=false, ResendScheduled=22:00:52.32, Timeout=22:00:51.849, PendingResendSkips=0, DeliveryState=unsent, Body=0x0000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000..., Body.length=1398}; requesting the departure confirmation for Member(Id=5, Timestamp=2012-07-30 15:35:09.735, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:21324, Role=ApacheCatalinaStartupBootstrap)
by MemberSet(Size=2, BitSetCount=2
Member(Id=1, Timestamp=2012-07-27 10:46:51.616, Address=10.34.32.101:8088, MachineId=2149, Location=machine:dev1ssapp3,process:1, Role=CoherenceServer)
Member(Id=12, Timestamp=2012-07-30 22:00:43.132, Address=10.34.32.101:8091, MachineId=2149, Location=machine:dev1ssapp3,process:7803, Role=ApacheCatalinaStartupBootstrap)
- Then the member set confirmed the departure however at the same time, service cluster also left.
2012-07-30 22:00:52.046/61.000 Oracle Coherence GE 3.5.1/461 <Info> (thread=Cluster, member=n/a): Member departure confirmed by MemberSet(Size=1, BitSetCount=2
Member(Id=12, Timestamp=2012-07-30 22:00:43.132, Address=10.34.32.101:8091, MachineId=2149, Location=machine:dev1ssapp3,process:7803, Role=ApacheCatalinaStartupBootstrap)
); removing Member(Id=5, Timestamp=2012-07-30 15:35:09.735, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:21324, Role=ApacheCatalinaStartupBootstrap)
2012-07-30 22:00:52.046/61.000 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=5, Timestamp=2012-07-30 22:00:52.046, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:21324, Role=ApacheCatalinaStartupBootstrap) left Cluster with senior member 1
2012-07-30 22:00:52.049/61.003 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster left the cluster
- Then the timeout during service start hence application fails to start
2012-07-30 22:00:52.051/61.005 Oracle Coherence GE 3.5.1/461 <Error> (thread=main, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=ServiceMemberSet(
OldestMember=n/a
ActualMemberSet=MemberSet(Size=15, BitSetCount=2
Member(Id=1, Timestamp=2012-07-27 10:46:51.616, Address=10.34.32.101:8088, MachineId=2149, Location=machine:dev1ssapp3,process:1, Role=CoherenceServer)
Member(Id=2, Timestamp=2012-07-27 10:47:12.122, Address=10.34.32.101:8089, MachineId=2149, Location=machine:dev1ssapp3,process:2, Role=CoherenceServer)
Member(Id=3, Timestamp=2012-07-27 10:48:02.603, Address=10.34.32.104:8088, MachineId=2152, Location=machine:dev1ssapp4,process:1, Role=CoherenceServer)
Member(Id=4, Timestamp=2012-07-27 10:48:04.76, Address=10.34.32.104:8089, MachineId=2152, Location=machine:dev1ssapp4,process:2, Role=CoherenceServer)
Member(Id=8, Timestamp=2012-07-30 14:27:07.382, Address=10.34.32.101:8090, MachineId=2149, Location=machine:dev1ssapp3,process:23727, Role=ApacheCatalinaStartupBootstrap)
Member(Id=9, Timestamp=2012-07-30 22:00:28.596, Address=10.34.32.101:8092, MachineId=2149, Location=machine:dev1ssapp3,process:7619, Role=ApacheCatalinaStartupBootstrap)
Member(Id=10, Timestamp=2012-07-30 14:34:27.573, Address=10.34.32.104:8090, MachineId=2152, Location=machine:dev1ssapp4,process:25219, Role=ApacheCatalinaStartupBootstrap)
Member(Id=11, Timestamp=2012-07-30 22:00:41.609, Address=10.34.32.107:8088, MachineId=2155, Location=machine:dev1sxapp2,process:17632, Role=ApacheCatalinaStartupBootstrap)
Member(Id=12, Timestamp=2012-07-30 22:00:43.132, Address=10.34.32.101:8091, MachineId=2149, Location=machine:dev1ssapp3,process:7803, Role=ApacheCatalinaStartupBootstrap)
Member(Id=14, Timestamp=2012-07-30 15:35:09.811, Address=10.34.32.106:8088, MachineId=2154, Location=machine:dev1sxapp1,process:5186, Role=ApacheCatalinaStartupBootstrap)
Member(Id=15, Timestamp=2012-07-30 16:02:34.096, Address=10.34.32.106:8091, MachineId=2154, Location=machine:dev1sxapp1,process:2691, Role=ApacheCatalinaStartupBootstrap)
Member(Id=16, Timestamp=2012-07-30 16:08:41.885, Address=10.34.32.107:8091, MachineId=2155, Location=machine:dev1sxapp2,process:15992, Role=ApacheCatalinaStartupBootstrap)
Member(Id=21, Timestamp=2012-07-30 21:58:56.669, Address=10.34.32.106:8089, MachineId=2154, Location=machine:dev1sxapp1,process:28689, Role=ApacheCatalinaStartupBootstrap)
Member(Id=22, Timestamp=2012-07-30 21:58:58.29, Address=10.34.32.107:8090, MachineId=2155, Location=machine:dev1sxapp2,process:15491, Role=ApacheCatalinaStartupBootstrap)
Member(Id=23, Timestamp=2012-07-30 22:00:21.648, Address=10.34.32.106:8090, MachineId=2154, Location=machine:dev1sxapp1,process:556, Role=ApacheCatalinaStartupBootstrap)
MemberId/ServiceVersion/ServiceJoined/ServiceLeaving
1/3.5/Fri Jul 27 10:46:51 EDT 2012/false,
2/3.5/Fri Jul 27 10:47:12 EDT 2012/false,
3/3.5/Fri Jul 27 10:48:02 EDT 2012/false,
4/3.5/Fri Jul 27 10:48:04 EDT 2012/false,
8/3.5/Mon Jul 30 14:27:07 EDT 2012/false,
9/3.5/Mon Jul 30 22:00:28 EDT 2012/false,
10/3.5/Mon Jul 30 14:34:27 EDT 2012/false,
11/3.5/Mon Jul 30 22:00:41 EDT 2012/false,
12/3.5/Mon Jul 30 22:00:43 EDT 2012/false,
14/3.5/Mon Jul 30 15:35:09 EDT 2012/false,
15/3.5/Mon Jul 30 16:02:34 EDT 2012/false,
16/3.5/Mon Jul 30 16:08:41 EDT 2012/false,
21/3.5/Mon Jul 30 21:58:56 EDT 2012/false,
22/3.5/Mon Jul 30 21:58:58 EDT 2012/false,
23/3.5/Mon Jul 30 22:00:21 EDT 2012/false
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:38)
     at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:366)
     at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
     at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
     at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:7)
     at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:27)
     at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
     at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:1011)
     at com.tangosol.coherence.hibernate.CoherenceCacheProvider.start(CoherenceCacheProvider.java:73)
     at org.hibernate.cache.impl.bridge.RegionFactoryCacheProviderBridge.start(RegionFactoryCacheProviderBridge.java:72)
Looking at member 5's log and I found it was being bounced at that time but somehow it failed to stop the coherence thread and didn't send out departure event to the cluster until was requested by other members.
SEVERE: The web application [riding-services] appears to have started a thread named Cluster but has failed to stop it. This is very likely to create a memory leak.
Questions:
1. Seems that this issue only happens when one server starts while another server is shut down at the same time range and both happen to be connected with each other for distributed caching. How can I modify the script to retry during startup when the first time it timed out? Or maybe modify the configuration to use a longer timeout value?
2. Is it possible to detect the unavailability of certain member quicker? Now seems 30 seconds or more.
Thanks in advance,

Similar Messages

My i pad is new at xmas it keeps disconnecting from internet my lap top is fine and ive tried resetting wi fi on i pad. can u help.

my i pad was brand new at xmas but keeps disconnecting from the internet. my lap top is fine and i have tried resetting the i pad. can u suggest anything else

Hi and thanks for your help, it was only my ipad that kept disconnecting from the internet but i think this happens when there are a lot of users on the same channel. it has been ok today so far, so, thanks again.

Why does my computer keep disconnecting from my wifi?

Why does my computer keep disconnecting from my wifi?

Some users with various Wi-Fi problems have reported that they were given the following instructions by Apple Support, sometimes with success. After completing the procedure, you'll have to recreate all your settings in the Networkpreference pane. Make sure you know how to do that before you begin. Taking screenshots of the preference pane may be helpful.
Step 1
Back up all data.
Triple-click anywhere in the line below on this page to select it:
/Library/Preferences/SystemConfiguration
Right-click or control-click the highlighted line and select
Services ▹ Reveal
from the contextual menu.* A folder should open with an item named "SystemConfiguration" selected. Move the selected item to the Trash. You may be prompted for your administrator password. Reboot and recreate the network settings.
*If you don't see the contextual menu item, copy the selected text to the Clipboard by pressing the key combinationcommand-C. In the Finder, select
Go ▹ Go to Folder...
from the menu bar and paste into the box that opens by pressing command-V. You won't see what you pasted because a line break is included. Press return.
Step 2
Reset the System Management Controller.
Step 3
Reset the PRAM.

Messages and FaceTime keep disconnecting from my MacBook Pro.

iMessage and FaceTime keep disconnecting from my MacBook Pro. I’ve never turned either of them off on my computer, but every time I turn them back on, they work for about a week, and then I start getting messages and phone calls on my iPhone and iPad but not on my MacBook. Is anyone else having this problem?

I'll ask the hosts to move this post to Yosemite.
Handoff Continuity Troubleshooting
iPhone, iPad, iPod touch, and Mac using Continuity - Connect

Why does my macbook pro keep disconnecting from the internet?

For whatever reason my Macbook pro 15in keeps disconnecting from the internet everytime it goes to sleep. And sometimes when I wake it up, I have to turn wifi on and off a couple times to get it to detect the right signal. This does not happen with my playstation or my phone so it has to be a problem with the Macbook itself. Can anyone give me any information on how to fix this problem? Anything would be greatly appreciated.

Hello, amal111.
Thank you for visiting Apple Support Communities.
Here is an article that I would recommend going through when experiencing Wi-Fi connectivity issues. Also, make sure all firmware updates have been processed on the router and the settings have been adjusted to the recommended settings in the second article below.
Wi-Fi: How to troubleshoot Wi-Fi connectivity
http://support.apple.com/kb/HT4628
iOS and OS X: Recommended settings for Wi-Fi routers and access points
http://support.apple.com/kb/HT4199
Cheers,
Jason H.

MAC book air keeps disconnecting from my wifi, any solutions?

My macbook air keeps disconnecting from my wifi. I also have a hp spectre and ipad air that have no wifi issues, any help?

Hello sandman1261999,
I'd suggest taking a look through the following article, as it provides numerous tips and information that can help stabilize your MacBook Air's WiFi connection.
Wi-Fi: How to troubleshoot Wi-Fi connectivity
http://support.apple.com/kb/HT4628
Cheers,
Allen

Apple tv keeps disconnecting from itunes

why does apple tv keep disconnecting from itunes

If on wifi try ethernet to rule out interference
You can also check www.istumbler.net
Also see
http://support.apple.com/kb/TS2972

Smart Watch 3 keeps disconnecting from phone

My swr50 keeps disconnecting from my motoG phone. I haven't figured out the circumstances for this...it seems random...I realise that I haven't had any notifications for a while, and none of the "OK Google" stuff works.My phone app says that it's connected, but it clearly ain't!

I wonder why it would disconnect that much,no problem like that on Z1. Do you have any task killers or antivirus?
What about stamina?
All we have to decide is what to do with the time that is given to us - J.R.R. Tolkien

Time capsule keeps disconnecting from the internet

My MBP was previously connected to the internet via my sky netgear router, with no issues.
I have recently purchased a 2TB time capsule and connected that to the sky netgear router via an ethernet cable.
I then set it up as its own wifi network. Now it keeps disconnecting from the internet every 5 mins or so. I know that the internet is fine as I can hook up with my netgear router with no issues. I cant seem to find any settings to play with. Earlier posts seem to suggest firmware issues, but since this is a brand new MBP and TC, I can't understand what the issue could be.

Give us the full blown picture of the network.
What modem? Is it a router?
Is the TC in router or bridge mode?
Can you test with ethernet.. does the same disconnect happen when wireless goes down?
ie is this a router issue or a wireless issue?
If wireless can you try setting all that is now auto, to manual settings.. wireless channels.. unfortunately I think wireless mode is no longer accessible. Different names for 2.4ghz and 5ghz.
There are no new drivers.. That would only be relevant to AC wireless anyway.. or the firmware of the TC.
But you are running the latest firmware.. 7.7.1 which does mention some (rare) disconnections.

My 21.5 inch desktop Mac keeps disconnecting from Wi-Fi - help?

My 21.5 inch desktop Mac keeps disconnecting from Wi-Fi. Help?

Read Wi-Fi: How to troubleshoot Wi-Fi connectivity
Please reply with how far you get with its suggestions.

HP Photosmart Premium C310a printer keeps disconnecting from my network.

My HP Photosmart Premium C310a printer keeps disconnecting from my network. I have tried disconnecting my Firewall and AVG virus software, all to no avail. Can anyone help?

This is an issue I have seen multiple times, and telling people to check their network settings, or to fix wireless settings does NOT address the root cause of the problem.   For those who are English-impared, here is a very clear explanation of what is going on:
The printer is REMOVED from devices and printers, seemingly at random.    This is NOT a problem with "The printer is not printing", where fixing the network or resetting IP addresses will fix it.   The printer is GONE, and needs to be added again.
I know networking, and at no time should a loss of connection cause the printer to just be removed from Printers and Faxes, Printers, or anything like that.    If the connection is lost, the connection is lost, but the printer SHOULD still be set up in Windows, just pointing at the wrong place(if the IP address has been changed).
I suspect that it is more likely that the HP Update software is the culprit, and will test that theory today with a customer who has this problem.    It also seems to be limited to the HP Photosmart line of printers, since the Officerjet line does not seem affected by the bug.

PC keeps disconnecting from home hub.

Got a very annoying problem, for some reason my computer keeps disconnecting from my home hub. Its got much worse recently and I have no clue what is doing it. Sometimes it can stay disconnected for a couple of hours and when you finally get the connection back it says it cant connect to the Internet. Also my network adaptor cant find my hub but it will find my neighbours even though the hub is just down the stairs from my PC. Signal it good-excellent. There are 2 other computers using the same hub but the are totally unaffected by this.
Would really appreciate some help.

from your comments I assume you are using wifi
you can try downloading inssider3 (istumbler if MAC) and then run it. This will show the broadcasting networks round about you and their channels including your own. If you then enter your router and change your wireless channel to a free or less congested channel.
If you like a post, or want to say thanks for a helpful answer, please click on the Ratings star on the left-hand side of the post.
If someone answers your question correctly please let other members know by clicking on ’Mark as Accepted Solution’.

Suddenly my Mac Air keeps disconnecting from wi fi. Help!!

My Mac Air keeps disconnecting from wifi. Help!!

Read Wi-Fi: How to troubleshoot Wi-Fi connectivity
Please reply with how far you get with its suggestions.

Why wireless apple keyboard keep disconnected from mac mini?

Why is my brand new wireless apple keyboard keep disconnected every 5 to 10 minutes from mac mini? the keyboard's battery is 100% full, and the bluetooth connection seems fine. I'm running 10.8.2. Please help...

Hi Barry,
You seem to have solutions ?
Do you know why iPhone 5 could not be paired by Bluetooth
with MacBook Pro Snow Leopard 10.6.8 ?

MacBook keep disconnecting from LinkSYS WAG325N WiFi Modem/Router

Hello,
I've been using my MacBook for a really long time. Over 1.5 years I had no Wi-Fi connection problems until my US Robotics 9106 Modem stopped working. I bought a nice LinkSYS WAG325N Modem. From that moment on, my MacBook started to disconnect from LinkSYS's WiFi constantly. I used every wep/web2 etc. security protocols possible. Nothing worked.
With my US Robotics I used MAC Filtering which worked perfectly. I did the same settings to LinkSYS as you can guess that didn't help at all...
I did firmware update to my modem, it even got worse so I downgrade my modem to previous version.
I read almost every forum out there, no one has came up with a solution.
If anyone has any idea how to fix this problem, please do so, I need urgent help or else I need to take my modem back before 15th day of using it is over. And try to find some modem that my MacBook would like...
Thank you.

Hello,
Yes I tried with US Robotics, worked perfectly.
It worked at all caffe's wifi spots (I have no info about their modem/router)
Strange enough, I realised that, if I place my old wifi modem next to this one just connecting the power and turning the modem on, somehow macbook connects fine !? it is too weird, and nearly impossible to explain. but if I place these two modems together and connect with linksys, and the other is just on, no connection no nothing just its wifi is on, MacBook connects... but if I leave the room and go to the living room the problem occurs again.
And since all the other windows laptops connecting from living room has %85 signal power this modem is wonderful. But for a MacBook, it is nothing but pain.
If I get a chance, I'll ask a friend to bring his macbook here which is newer (core2duo) mine is core duo and see what happens.

Urgent! Node keep disconnecting from Coherence Cluster

Similar Messages

Maybe you are looking for