EM for Coherence - Cluster upgrade
Hi Gurus,
I noticed that there are "Coherence Node Provisioning" process in EM12c, and it says "You can also update selected nodes by copying configuration files and restarting the nodes.". Does the EM will internally check the service HA status before update (stop/start) each node? The "NODE-SAFE" should be the minimum HA Status criterion to meet to ensure there is no data loss.
Thanks in advance
Hysun
Thanks for your hints, but it didn't work either. Maybe because the metaset uses the disks DID-name and those are not available when the node is not booted as part of the cluster.
What I hope will work is this:
- deactivate the zones resourcegroup
- make a backup of the non-global zones root
- restore the backup to a temporary filesystem on the nodes bootdisk
- mount the temporary filesystem as the zones root (via vfstab)
- upgrade this node including the zone
- reboot as part of the cluster (the zone should not start because of autoboot=false and the RG being deactivated)
- acquire access to the zones shared disk resource
- copy the content of the zones root back to its original place
- activate the zones resourcegroup
- upgrade the other node
- and of cource backups, backups and even more backups at the right moments :-)
I will test this scenario as soon as I can find the time for it. If I am successful I will post again.
Regards, Paul
Similar Messages
-
What are best practices for rolling out cluster upgrade?
Hello,
I am looking for your input on the approaches to implement Production RAC upgrade without having a spare instance of RAC servers. We have a 2-node database RAC 11.1 on Production. We are planning to upgrade to 11.2 but only have a single database that the pre-Production integration can be verified on. Our concern is that the integration may behave differently on RAC vs. the signle database instance. How everybody else approaches this problem?you want to test RAC upgrade on NON RAC database. If you ask me that is a risk but it depends on may things
Application configuration - If your application is configured for RAC, FAN etc. you cannot test it on non RAC systems
Cluster upgrade - If your standalone database is RAC one node you can probably test your cluster upgrade there. If you have non RAC database then you will not be able to test cluster upgrade or CRS
Database upgrade - There are differences when you upgrade RAC vs non RAC database which you will not be able to test
I think the best way for you is to convert your standalone database to RAC one node database and test it. that will take you close to multi node RAC -
Accessing Coherence Extend* Proxy Deployoed on Weblogic Coherence Cluster from Java Client
Hi,
I am trying to access Extend Proxy through Thick Java Client
Followed steps as per below links and deployed a GAR on 3 Server ( 2 Storage Enabled Coherence Cluster and 1 Coherence Storage Disabled Extend Proxy Enabled). I could see ExtendProxyService using JMX and can see Port running on the System.
Ref :
Setting Up Coherence*Extend - 12c (12.1.2)
http://docs.oracle.com/middleware/1212/coherence/COHAG/deploy_options.htm#CHDJBJDI
Issue :
When I tried to Execute Java Client to Connect to Proxy Server it Connects to Port and then Disconnects with ConnectionException as below.
Observer below Lines in Box is show he Connected Socket with Port 9099 which is Extend Proxy Port
Error Message
2013-11-08 14:55:55.114/1.202 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=TcpClientRemoteService:TcpInitiator, member=n/a): Started: TcpInitiator{Name=TcpClientRemoteService:TcpInitiator, State=(SERVICE_STARTED), ThreadCount=0, Codec=Codec(Format=POF), Serializer=com.tangosol.io.DefaultSerializer, PingInterval=0, PingTimeout=30000, RequestTimeout=30000, ConnectTimeout=10000, SocketProvider=[email protected], RemoteAddresses=WrapperSocketAddressProvider{Providers=[[DTC37446E9C6CBD/127.0.0.0:9099]]}, SocketOptions{LingerTimeout=0, KeepAliveEnabled=true, TcpDelayEnabled=false}}
2013-11-08 14:55:55.146/1.234 Oracle Coherence GE 12.1.2.0.0 <D5> (thread=main, member=n/a): Connecting Socket to 127.0.0.0:9099
2013-11-08 14:55:55.146/1.234 Oracle Coherence GE 12.1.2.0.0 <Info> (thread=main, member=n/a): Connected Socket to 127.0.0.0:9099
2013-11-08 14:55:55.161/1.249 Oracle Coherence GE 12.1.2.0.0 <Info> (thread=main, member=n/a): Error establishing a connection with 127.0.0.0:9099: com.tangosol.net.messaging.ConnectionException: TcpConnection(Id=null, Open=true, LocalAddress=0.0.0.0:54384, RemoteAddress=127.0.0.0:9099)
2013-11-08 14:55:55.161/1.249 Oracle Coherence GE 12.1.2.0.0 <Error> (thread=main, member=n/a): Error while starting service "TcpClientRemoteService": com.tangosol.net.messaging.ConnectionException: could not establish a connection to one of the following addresses: [127.0.0.0:9099]; make sure the "remote-addresses" configuration element contains an address and port of a running TcpAcceptor
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.initiator.TcpInitiator.openConnection(TcpInitiator.CDB:121)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.Initiator.ensureConnection(Initiator.CDB:11)
at com.tangosol.coherence.component.net.extend.remoteService.RemoteCacheService.openChannel(RemoteCacheService.CDB:7)
at com.tangosol.coherence.component.net.extend.RemoteService.doStart(RemoteService.CDB:11)
at com.tangosol.coherence.component.net.extend.RemoteService.start(RemoteService.CDB:5)
at com.tangosol.coherence.component.util.SafeService.startService(SafeService.CDB:53)
at com.tangosol.coherence.component.util.safeService.SafeCacheService.startService(SafeCacheService.CDB:5)
at com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:27)
at com.tangosol.coherence.component.util.SafeService.start(SafeService.CDB:14)
at com.tangosol.net.ExtensibleConfigurableCacheFactory.startService(ExtensibleConfigurableCacheFactory.java:681)
at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureService(ExtensibleConfigurableCacheFactory.java:599)
at com.tangosol.coherence.config.scheme.AbstractCachingScheme.realizeCache(AbstractCachingScheme.java:50)
at com.tangosol.coherence.config.scheme.AbstractBundlingScheme.realizeCache(AbstractBundlingScheme.java:31)
at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureCache(ExtensibleConfigurableCacheFactory.java:254)
at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:205)
at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:182)If this proxy design (not starting up due to a invalid entry in "authroized-hosts") is on-purpose from Coherence Engineers - then it should be re-visited.
I think the PROXY Server should just log a message stating about the invalid DNS entry for the Authorized-host and continue with the startup...Failing to start completely doesn;t make sense since one cannot rely completely on DNS to
say everything should be correct before a server start.
Ofcourse you can overcome by writing your own Custom Filter - but the issue pop's out as with any custom filter(s) is maintaining them along the road (with all minor/major coherence upgrades).
Also - this "Authorized-Hosts" concept should be carefully analyzed particularly for the following issues...
(1) if the client IP is changed in the DNS server - will the proxy-server allow the new Client connection without any issues? when will the PROXY server flush its CLIENT DNS entries or what is the TTL time-limit for a CLIENT cached through Authorized-hosts by the PROXY-SERVER?
(2) Suppose, we have a CLIENT in the "Authroized-Hosts" making a valid connection to the PROXY and putting some cache into the SERVER CACHE through the PROXY....now if the IP-address (DNS being the same) of the CLIENT is changed - can the CLIENT can GET the CACHE it just PUT into the SERVER without any ERRORS?
(3) How often we need to re-start PROXIES? Do we need to re-start them often for the DNS issues (if any) mentioned above?
Looks like the Limited documentation & examples for Coherence*Extend - particularly for .NET & C++ clients & *Extend Proxies is a point of concern.
vk -
Creating sub-cluster within a Coherence cluster
Hi all,
Does Coherence support creation of 'sub-clusters' within a larger coherence cluster - such that certain caches can be configured to run only on these subclusters, and other caches run on the entire coherence cluster as usual.
E.g., suppose my application consists of 3 websphere clusters (under same cell) - each cluster consists of 2 Websphere server instances. Each Websphere cluster has got a specific functional responsbility (e.g., 1 cluster handles the UI, one handles core processing functionality and the 3rd cluster handles links with external legacy systems). Since the functionality itself is 'partitioned' - its possible that certain data managed by a particular WAS cluster should only be managed within that cluster and not across all 6 WAS instances.
So - in this case - suppose I do have an 'outer' Coherence cluster of all 6 WAS instances (and some Caches are configured to be acessible to all 6 WAS instances - since the data managed in these caches is needed by all 6 WAS instances). Can I configure a smaller Coherence cluster to be available only on say 2 of the Websphere instances (say the WAS cluster which handles legacy links) - and configure certain caches which are available only on this smaller sub-cluster.
regards,
Sanjeev.I am quite confused about the purpose of the service-name. How would you tie down a cache to a particular service? In the context of the above example, the requirement seems to be:
CacheA should be spread over the UI cluster.
CacheB should be spread over the legacy cluster.
CacheC should be spread over the global cluster.
Are you suggesting something like the following:
<u>Cache config file on a UI node</u>:
<cluster-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>CacheA</cache-name>
<scheme-name>ui</scheme-name>
</cache-mapping>
<cache-mapping>
<cache-name>CacheC</cache-name>
<scheme-name>global</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>ui</scheme-name>
<service-name>ui</service-name>
</distributed-scheme>
<distributed-scheme>
<scheme-name>global</scheme-name>
<service-name>global</service-name>
</distributed-scheme>
</caching-schemes>
</cluster-config><u>Cache config file on a legacy node</u>:
<cluster-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>CacheB</cache-name>
<scheme-name>legacy</scheme-name>
</cache-mapping>
<cache-mapping>
<cache-name>CacheC</cache-name>
<scheme-name>global</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>legacy</scheme-name>
<service-name>legacy</service-name>
</distributed-scheme>
<distributed-scheme>
<scheme-name>global</scheme-name>
<service-name>global</service-name>
</distributed-scheme>
</caching-schemes>
</cluster-config>The basic question seems to be: how do you control the nodes over which a cache is spread, purely from the cache name?
Also, the 3.2 <role-name> feature seems to be something that addresses this requirement. How does that play v/s a service-name?
My requirement is similar (needing to control the nodes over which different caches are spread) but I do not quite understand how the service-name would be used to satisfy this example. Could you please explain via cache configurations for this example?
Thanks
Ghanshyam -
What's the maximum size of data a coherence cluster can hold?
What's the maximum size of data a coherence cluster can hold before it starts noticing a degradation in performance?
Assume a partitioned topology is used with only one backup for each partition.Hi,
Coherence partitioned cache is designed for linear scalability and it does it quite well. I don't see any reasons of performance degrations with increase in data size given, you have enough cores and memory for processing the requests and managing the data.
Cheers,
_NJ -
Coherence Cluster Errors- Need your help to solve
Hi,
We had this error recently in QA and these servers are not new servers. These servers were running from some time and in good condition.
We had a below error happened suddently and cuased servers outage for some time.
After restarted all the servers, this issue has gone.
We are trying to understand the root cause to avoid this issue in future and need expertise in this forum for that.
Brief summary of issue
1. We had performed multicaste testing on the coherence cluster IP and port and all the communication is good.
2. Issues started with error of Unable to refresh sockets:
Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: Unable to refresh sockets: [UnicastUdpSocket{State=STATE_OPEN, address:port=1.1.1.85:8088}, MulticastUdpSocket{State=STATE_OPEN, address:port=239.3.1.17:35122, InterfaceAddress=10.137.3.85, TimeToLive=1}, TcpSocketAccepter{State=STATE_OPEN, ServerSocket=1.1.1.85:8088}]; last failed socket: MulticastUdpSocket{State=STATE_OPEN, address:port=239.3.1.17:35122, InterfaceAddress=10.137.3.85, TimeToLive=1}
at com.tangosol.coherence.component.net.Cluster$SocketManager.refreshSockets(Cluster.CDB:91)
at com.tangosol.coherence.component.net.Cluster$SocketManager$MulticastUdpSocket.onInterruptedIOException(Cluster.CDB:9)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:33)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: Receive timed out
3. After that, I noticed copule of errors like
Restarting Service: DistributedCache validatePolls: This service timed-out due to unanswered handshake request. Manual intervention is required to stop the members that have not responded to this Poll
4. Continously logging errors like : Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/
5. After that noticed,
Service DistributedCache: received ServiceConfigSync containing 272 entries
2013-10-26 08:26:43,241 -0700 level=ERROR class="STDERR" 2013-10-26 08:26:43.241/76.243 Oracle Coherence GE 3.5.1/461 <Error> (thread=main, member=1): Error while starting service "DistributedCache": com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=2, Name=DistributedCache, Type=DistributedCache
MemberSet=ServiceMemberSet(
OldestMember=Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=10.137.3.49:8088, MachineId=32817, Location=site:test.test.net,machine:test30b,process:3870)
ActualMemberSet=MemberSet(Size=3, BitSetCount=2
Member(Id=1, Timestamp=2013-10-26 08:26:12.289, Address=1.1.1.85:8088, MachineId=32853, Location=site:test.test.net,machine:test304,process:6207, Role=JavaLangThread)
Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1.49:8088, MachineId=32817, Location=site:test.test.net,machine:test30b,process:3870)
Member(Id=5, Timestamp=2013-10-26 08:26:29.871, Address=1.1.1.86:8088, MachineId=32854, Location=site:test.test.net,machine:test305,process:3988)
MemberId/ServiceVersion/ServiceJoined/ServiceLeaving
1/3.5/Sat Oct 26 08:26:13 PDT 2013/false,
3/3.5/Sat Oct 26 05:16:47 PDT 2013/false,
5/3.5/Sat Oct 26 08:26:30 PDT 2013/false
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
Your Help is highly appreciated !!!!
Detailed Server Error Log:
2013-10-26 00:15:13,280 -0700 level=ERROR class="STDERR"
2013-10-26 00:15:13.279/2079180.072 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 2642 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread); 34 packets rescheduled, PauseRate=0.0010, Threshold=222
2013-10-26 00:15:15,508 -0700 level=ERROR class="STDERR"
2013-10-26 00:15:15.508/2079182.301 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 4875 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2013-10-08 22:00:17.258, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Role=JavaLangThread); 47 packets rescheduled, PauseRate=3.0E-4, Threshold=1438
2013-10-26 01:15:29,028 -0700 level=ERROR class="STDERR"
2013-10-26 01:15:29.018/2082795.811 Oracle Coherence GE 3.5.1/461 <Info> (thread=PacketListenerN, member=4): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2013-10-26 01:15:29,036 -0700 level=ERROR class="STDERR"
2013-10-26 01:15:29.036/2082795.829 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 13068 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2013-10-08 22:00:17.258, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Role=JavaLangThread); 86 packets rescheduled, PauseRate=4.0E-4, Threshold=1438
2013-10-26 01:15:29,037 -0700 level=ERROR class="STDERR"
2013-10-26 01:15:29.036/2082795.829 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 13069 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread); 84 packets rescheduled, PauseRate=0.0010, Threshold=269
2013-10-26 01:31:44,494 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 02:15:34,907 -0700 level=ERROR class="STDERR"
2013-10-26 02:15:34.906/2086401.699 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 6476 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread); 24 packets rescheduled, PauseRate=0.0011, Threshold=313
2013-10-26 02:43:52,199 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 03:00:55,493 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 03:15:41,144 -0700 level=ERROR class="STDERR"
2013-10-26 03:15:41.144/2090007.937 Oracle Coherence GE 3.5.1/461 <D5> (thread=PacketPublisher, member=4): Experienced a 202 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2013-10-08 22:00:17.258, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Role=JavaLangThread); 25 packets rescheduled, PauseRate=4.0E-4, Threshold=1509
2013-10-26 03:15:41,592 -0700 level=ERROR class="STDERR"
2013-10-26 03:15:41.592/2090008.385 Oracle Coherence GE 3.5.1/461 <D5> (thread=PacketPublisher, member=4): Experienced a 371 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread); 41 packets rescheduled, PauseRate=0.0010, Threshold=290
2013-10-26 03:31:38,099 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 04:15:47,869 -0700 level=ERROR class="STDERR"
2013-10-26 04:15:47.869/2093614.662 Oracle Coherence GE 3.5.1/461 <D5> (thread=PacketPublisher, member=4): Experienced a 850 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2013-10-08 22:00:17.258, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Role=JavaLangThread); 52 packets rescheduled, PauseRate=4.0E-4, Threshold=1509
2013-10-26 04:16:00,192 -0700 level=ERROR class="STDERR"
2013-10-26 04:16:00.182/2093626.975 Oracle Coherence GE 3.5.1/461 <Info> (thread=PacketListenerN, member=4): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2013-10-26 04:16:00,199 -0700 level=ERROR class="STDERR"
2013-10-26 04:16:00.199/2093626.992 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 13180 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread); 126 packets rescheduled, PauseRate=0.0011, Threshold=424
2013-10-26 04:16:01,897 -0700 level=ERROR class="STDERR"
2013-10-26 04:16:01.897/2093628.690 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 1503 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2013-10-08 22:00:17.258, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Role=JavaLangThread); 173 packets rescheduled, PauseRate=4.0E-4, Threshold=1509
2013-10-26 04:26:54,424 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 04:51:52,096 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:02:52,292 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:16:06,076 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.075/2097232.868 Oracle Coherence GE 3.5.1/461 <Error> (thread=PacketListenerN, member=4):
Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: Unable to refresh sockets: [UnicastUdpSocket{State=STATE_OPEN, address:port=1.1.1..85:8088}, MulticastUdpSocket{State=STATE_OPEN, address:port=239.3.1.17:35122, InterfaceAddress=1.1.1..85, TimeToLive=1}, TcpSocketAccepter{State=STATE_OPEN, ServerSocket=1.1.1..85:8088}]; last failed socket: MulticastUdpSocket{State=STATE_OPEN, address:port=239.3.1.17:35122, InterfaceAddress=1.1.1..85, TimeToLive=1}
at com.tangosol.coherence.component.net.Cluster$SocketManager.refreshSockets(Cluster.CDB:91)
at com.tangosol.coherence.component.net.Cluster$SocketManager$MulticastUdpSocket.onInterruptedIOException(Cluster.CDB:9)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:33)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: Receive timed out
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
at java.net.DatagramSocket.receive(DatagramSocket.java:725)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
2013-10-26 05:16:06,080 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.080/2097232.873 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=4): Service Cluster left the cluster
2013-10-26 05:16:06,105 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.105/2097232.898 Oracle Coherence GE 3.5.1/461 <D5> (thread=Invocation:Management, member=4): Service Management left the cluster
2013-10-26 05:16:06,105 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.105/2097232.898 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-180, member=4): Restarting NamedCache: test234aaaapeu-cache
2013-10-26 05:16:06,105 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.105/2097232.898 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-180, member=4): Restarting Service: DistributedCache
2013-10-26 05:16:06,110 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.106/2097232.899 Oracle Coherence GE 3.5.1/461 <Error> (thread=DistributedCache, member=4):
validatePolls: This service timed-out due to unanswered handshake request. Manual intervention is required to stop the members that have not responded to this Poll
PollId=24209529, active
InitTimeMillis=1382789736843
Service=DistributedCache (2)
RespondedMemberSet=[]
LeftMemberSet=[]
RemainingMemberSet=[3]
Request=Message "LockRequest"
{test.test.net
FromMember=Member(Id=4, Timestamp=2013-10-24 15:16:09.067, Address=1.1.1..85:8088, MachineId=32853, Location=site:test.test.net,machine:testabc304,process:4000)
FromMessageId=38338332
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=12
ToPollId=0
Poll=null
Packets
Service=DistributedCache{Name=DistributedCache, State=(SERVICE_STOPPED), Not initialized}
ToMemberSet=MemberSet(Size=1, BitSetCount=1
Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread)
NotifySent=false
null
WaitTimeout=1382789776739, LeaseExpiration=9223372036854775807
2013-10-26 05:16:06,110 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.109/2097232.902 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=4): Service DistributedCache left the cluster
2013-10-26 05:16:06,117 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.117/2097232.910 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-180, member=n/a): Restarting cluster
2013-10-26 05:16:06,198 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.198/2097232.991 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2013-10-26 05:16:07,410 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.410/2097234.203 Oracle Coherence GE 3.5.1/461 <Info> (thread=Cluster, member=n/a): Created a new cluster "cluster:0x27CB" with Member(Id=1, Timestamp=2013-10-26 05:16:06.128, Address=1.1.1..85:8088, MachineId=32853, Location=site:test.test.net,machine:testabc304,process:4000, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4) UID=0x0A89035500000141F4B15BF080551F98
2013-10-26 05:16:07,436 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.436/2097234.229 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-180, member=1): Restarting Service: Management
2013-10-26 05:16:07,450 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.450/2097234.243 Oracle Coherence GE 3.5.1/461 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
2013-10-26 05:16:07,474 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.474/2097234.267 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 1
2013-10-26 05:16:07,491 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.491/2097234.284 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-183, member=1): Restarting NamedCache: test234aaaaficustomer-cache
2013-10-26 05:16:07,514 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.514/2097234.307 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-38, member=1): Restarting NamedCache: test234aaaaaccount-no-export-cache
2013-10-26 05:16:07,529 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.529/2097234.322 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-136, member=1): Restarting NamedCache: test234aaaausrsum-cache
2013-10-26 05:16:07,546 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.545/2097234.338 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-9, member=1): Restarting NamedCache: test234aaaafi-v2-cache
2013-10-26 05:16:07,569 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.567/2097234.360 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-59, member=1): Restarting NamedCache: test234aaaaaccount-v2-cache
2013-10-26 05:16:07,748 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.748/2097234.541 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-28, member=1): Restarting NamedCache: test234aaaafi-cache
2013-10-26 05:16:07,816 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.816/2097234.609 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-133, member=1): Restarting NamedCache: test234aaaahistory-v2-cache
2013-10-26 05:16:09,154 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.154/2097235.947 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-134, member=1): Restarting NamedCache: test234aaaaaccount-cache
2013-10-26 05:16:09,169 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.169/2097235.962 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-134, member=1): Restarting NamedCache: test234aaaahistory-cache
2013-10-26 05:16:09,444 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.444/2097236.237 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2013-10-26 05:16:09.259, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988) joined Cluster with senior member 1
2013-10-26 05:16:09,539 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.539/2097236.332 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 2 joined Service Management with senior member 1
2013-10-26 05:16:09,580 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.579/2097236.372 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 2 joined Service DistributedCache with senior member 1
2013-10-26 05:16:09,599 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.599/2097236.392 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Service DistributedCache: sending ServiceConfigSync containing 268 entries to Member 2
2013-10-26 05:16:09,681 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.681/2097236.474 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 1> Transferring 128 out of 257 vulnerable partitions to member 2 requesting 128
2013-10-26 05:16:09,892 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.881/2097236.674 Oracle Coherence GE 3.5.1/461 <D4> (thread=DistributedCache, member=1): 1> Transferring 129 out of 129 partitions to a machine-safe backup 1 at member 2 (under 129)
2013-10-26 05:16:09,901 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.901/2097236.694 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 388KB of backup[1] for PartitionSet{128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256} to member 2
2013-10-26 05:16:10,415 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:10.415/2097237.208 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): TcpRing: connecting to member 2 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..86,port=8088,localport=37005]}
2013-10-26 05:16:10,657 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:10.657/2097237.450 Oracle Coherence GE 3.5.1/461 <Warning> (thread=Cluster, member=1): Received panic from junior member Member(Id=2, Timestamp=2013-10-26 05:16:09.259, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988) caused by Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread)
2013-10-26 05:16:11,592 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:11.592/2097238.385 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32822,localport=8088]}
2013-10-26 05:16:13,568 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:13.568/2097240.361 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-52, member=1): Restarting NamedCache: test234aaaauserData-cache
2013-10-26 05:16:13,596 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:13.596/2097240.389 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32823,localport=8088]}
2013-10-26 05:16:14,937 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:14.937/2097241.730 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-52, member=1): Restarting NamedCache: test234aaaacheckimage-cache
2013-10-26 05:16:15,600 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:15.600/2097242.393 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32824,localport=8088]}
2013-10-26 05:16:17,602 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:17.602/2097244.395 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32825,localport=8088]}
2013-10-26 05:16:19,605 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:19.605/2097246.398 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32828,localport=8088]}
2013-10-26 05:16:21,609 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:21.609/2097248.402 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32829,localport=8088]}
2013-10-26 05:16:23,611 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:23.611/2097250.404 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32830,localport=8088]}
2013-10-26 05:16:25,616 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:25.616/2097252.409 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32831,localport=8088]}
2013-10-26 05:16:27,619 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:27.619/2097254.412 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32832,localport=8088]}
2013-10-26 05:16:29,621 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:29.621/2097256.414 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32833,localport=8088]}
2013-10-26 05:16:31,626 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:31.626/2097258.419 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32834,localport=8088]}
2013-10-26 05:16:33,631 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:33.631/2097260.424 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32835,localport=8088]}
2013-10-26 05:16:35,632 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:35.632/2097262.425 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32836,localport=8088]}
2013-10-26 05:16:37,636 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:37.635/2097264.428 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32837,localport=8088]}
2013-10-26 05:16:39,641 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:39.640/2097266.433 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32838,localport=8088]}
2013-10-26 05:16:41,643 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:41.643/2097268.436 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32841,localport=8088]}
2013-10-26 05:16:47,329 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:47.329/2097274.122 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870) joined Cluster with senior member 1
2013-10-26 05:16:47,425 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:47.425/2097274.218 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 3 joined Service Management with senior member 1
2013-10-26 05:16:47,477 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:47.476/2097274.269 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 3 joined Service DistributedCache with senior member 1
2013-10-26 05:16:47,501 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:47.500/2097274.294 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Service DistributedCache: sending ServiceConfigSync containing 270 entries to Member 3
2013-10-26 05:16:47,548 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:47.548/2097274.341 Oracle Coherence GE 3.5.1/461 <D5> (thread=TcpRingListener, member=1): TcpRing: connecting to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32846,localport=8088]}
2013-10-26 05:16:48,454 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:48.453/2097275.246 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 43 out of 129 primary partitions to member 3 requesting 43
2013-10-26 05:16:48,709 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:48.709/2097275.502 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 39 out of 125 primary partitions to member 3 requesting 39
2013-10-26 05:16:48,885 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:48.884/2097275.677 Oracle Coherence GE 3.5.1/461 <D5> (thread=http-0.0.0.0-8080-210, member=1): Repeating QueryRequest due to the re-distribution of PartitionSet{132, 133, 134, 135, 136, 137, 138, 139, 140, 141}
2013-10-26 05:16:50,850 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:50.848/2097277.641 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 29 out of 115 primary partitions to member 3 requesting 29
2013-10-26 05:16:50,968 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:50.968/2097277.761 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 21 out of 107 primary partitions to member 3 requesting 21
2013-10-26 05:16:51,097 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.097/2097277.890 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 14 out of 100 primary partitions to member 3 requesting 14
2013-10-26 05:16:51,218 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.218/2097278.011 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 6 out of 92 primary partitions to member 3 requesting 6
2013-10-26 05:16:51,340 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.340/2097278.133 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 1 out of 87 primary partitions to member 3 requesting 1
2013-10-26 05:16:51,352 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.352/2097278.145 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 540KB of backup[1] for PartitionSet{171, 172, 173, 174, 175, 176, 177} to member 3
2013-10-26 05:16:51,465 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.464/2097278.257 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 575KB of backup[1] for PartitionSet{178, 179, 180, 181, 182, 183} to member 3
2013-10-26 05:16:51,569 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.569/2097278.362 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 537KB of backup[1] for PartitionSet{184, 185, 186, 187} to member 3
2013-10-26 05:16:51,688 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.688/2097278.481 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 553KB of backup[1] for PartitionSet{188, 189, 190, 191, 192, 193, 194, 195, 196} to member 3
2013-10-26 05:16:51,817 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.817/2097278.610 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 526KB of backup[1] for PartitionSet{197, 198, 199, 200, 201, 202} to member 3
2013-10-26 05:16:51,928 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.928/2097278.721 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 768KB of backup[1] for PartitionSet{203, 204, 205, 206, 207, 208, 209} to member 3
2013-10-26 05:16:52,040 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:52.039/2097278.832 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 198KB of backup[1] for PartitionSet{210, 211, 212, 213} to member 3
2013-10-26 05:19:06,157 -0700 level=ERROR class="STDERR"
2013-10-26 05:19:06.157/2097412.950 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-63, member=1): Restarting NamedCache: throttleData-cache
2013-10-26 05:22:15,094 -0700 level=ERROR class="STDERR"
2013-10-26 05:22:15.094/2097601.887 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-136, member=1): Restarting NamedCache: test234aaaadepositslipimage-cache
2013-10-26 05:22:17,183 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:28:49,617 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:29:39,729 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:33:37,607 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:39:33,872 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 06:49:30,617 -0700 level=ERROR class="STDERR"
2013-10-26 06:49:30.617/2102837.410 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=1): Experienced a 6378 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-26 05:16:09.259, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988); 56 packets rescheduled, PauseRate=0.0011, Threshold=1976
2013-10-26 07:39:18,855 -0700 level=ERROR class="STDERR"
2013-10-26 07:39:18.854/2105825.647 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=1): Experienced a 7318 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870); 68 packets rescheduled, PauseRate=8.0E-4, Threshold=497
2013-10-26 07:49:37,510 -0700 level=ERROR class="STDERR"
2013-10-26 07:49:37.510/2106444.303 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=1): Experienced a 6653 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-26 05:16:09.259, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988); 69 packets rescheduled, PauseRate=0.0014, Threshold=1785
Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
2013-10-26 08:26:11,291 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:11.291/44.293 Oracle Coherence GE 3.5.1/461 <Info> (thread=main, member=n/a): Loaded cache configuration from "file:/usr/local/whp-jboss-web-5/server/default/env/test234aaaacoherence-cache-config.xml"
2013-10-26 08:26:12,263 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.263/45.265 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2013-10-26 08:26:12,477 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.477/45.479 Oracle Coherence GE 3.5.1/461 <Info> (thread=Cluster, member=n/a): This Member(Id=1, Timestamp=2013-10-26 08:26:12.289, Address=1.1.1..85:8088, MachineId=32853, Location=site:test.test.net,machine:testabc304,process:6207, Role=JavaLangThread, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4) joined cluster "cluster:0x27CB" with senior Member(Id=2, Timestamp=2013-10-26 05:16:09.259, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4)
2013-10-26 08:26:12,501 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.501/45.503 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870) joined Cluster with senior member 2
2013-10-26 08:26:12,507 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.506/45.508 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 2 joined Service Management with senior member 2
2013-10-26 08:26:12,507 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.507/45.509 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 2 joined Service DistributedCache with senior member 2
2013-10-26 08:26:12,520 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.520/45.522 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 3 joined Service Management with senior member 2
2013-10-26 08:26:12,520 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.520/45.522 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 3 joined Service DistributedCache with senior member 2
2013-10-26 08:26:12,639 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.639/45.641 Oracle Coherence GE 3.5.1/461 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 2
2013-10-26 08:26:12,700 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.700/45.702 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): TcpRing: connecting to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=8088,localport=52891]}
2013-10-26 08:26:13,191 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:13.190/46.193 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 2
2013-10-26 08:26:14,538 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:14.538/47.540 Oracle Coherence GE 3.5.1/461 <D5> (thread=TcpRingListener, member=1): TcpRing: connecting to member 2 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..86,port=40281,localport=8088]}
2013-10-26 08:26:29,695 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:29.694/62.696 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): TcpRing: disconnected from member 2 due to a kill request
2013-10-26 08:26:29,695 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:29.694/62.696 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 2 left service Management with senior member 3
2013-10-26 08:26:29,695 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:29.694/62.696 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 2 left service DistributedCache with senior member 3
2013-10-26 08:26:29,696 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:29.696/62.698 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2013-10-26 08:26:29.694, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988) left Cluster with senior member 3
2013-10-26 08:26:30,069 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:30.069/63.071 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member(Id=5, Timestamp=2013-10-26 08:26:29.871, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988) joined Cluster with senior member 3
2013-10-26 08:26:30,271 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:30.271/63.273 Oracle Coherence GE 3.5.1/461 <D5> (thread=TcpRingListener, member=1): TcpRing: connecting to member 5 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..86,port=40285,localport=8088]}
2013-10-26 08:26:30,272 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:30.272/63.274 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 5 joined Service Management with senior member 3
2013-10-26 08:26:30,443 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:30.443/63.445 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 5 joined Service DistributedCache with senior member 3
2013-10-26 08:26:38,739 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:38.738/71.740 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Service DistributedCache: received ServiceConfigSync containing 272 entries
2013-10-26 08:26:43,241 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:43.241/76.243 Oracle Coherence GE 3.5.1/461 <Error> (thread=main, member=1): Error while starting service "DistributedCache": com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=2, Name=DistributedCache, Type=DistributedCache
MemberSet=ServiceMemberSet(
OldestMember=Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870)
ActualMemberSet=MemberSet(Size=3, BitSetCount=2
Member(Id=1, Timestamp=2013-10-26 08:26:12.289, Address=1.1.1..85:8088, MachineId=32853, Location=site:test.test.net,machine:testabc304,process:6207, Role=JavaLangThread)
Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870)
Member(Id=5, Timestamp=2013-10-26 08:26:29.871, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988)
MemberId/ServiceVersion/ServiceJoined/ServiceLeaving
1/3.5/Sat Oct 26 08:26:13 PDT 2013/false,
3/3.5/Sat Oct 26 05:16:47 PDT 2013/false,
5/3.5/Sat Oct 26 08:26:30 PDT 2013/false
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:38)
at com.tangosol.coherence.component.util.SafeService.startService(SafeService.CDB:28)
at com.tangosol.coherence.component.util.safeService.SafeCacheService.startService(SafeCacheService.CDB:5)
at com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:27)
at com.tangosol.coherence.component.util.SafeService.start(SafeService.CDB:14)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:973)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:842)
at com.tangosol.net.DefaultConfigurableCacheFactory.configureCache(DefaultConfigurableCacheFactory.java:1053)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:290)
at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:747)
at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:724Hi
The common causes of communication delays and packet timeouts are excessive GC pauses, high CPU usage, and swapping.
Each of these occurrences may disrupt the Coherence packet processing threads, thus preventing the processing and acknowledgment of packets from other cluster members.
1 check GC performance , see process memory consumption and GC logs.
2 check cpu , vmstat , top command.
3 check swap , vmstat command.
see Oracle Support Doc ID 1110544.1
Although communication delays and packet timeouts can be caused by network related issue.
check performance network :
Performing a Datagram Test for Network Performance - Coherence 3.5 User Guide - Oracle Coherence Knowledge Base
regards,
Leo_TA -
Java Application connectivity to Coherence cluster
I am looking around the options for Java application connecting to Coherence cluster..
These are the options I seem to have
1. Coherence Extend -
2. Make Java application as a cluster member with no storage
what are the pros/cons?
ThanksHi,
Here are my thought off the top of my head
*1. Use Extend if...*
<li> Your application is short lived - i.e. does not run for very long (minutes)
<li> Your application is not located close (in network terms) to the rest of the cluster
<li> Your client uses a different version of Coherence to the cluster (there are caveats around this)
<li> The client application is being developed by another team and its release cycle is not under your control
Pros:
Good for short lived applications
Good for remotely located clients
Can be written in other languages than Java
Cons:
Extend will perform slower than a cluster member as every request goes via the proxy server and then to the rest of the cluster.
If using an invocation service where you want to run invocables across the cluster it needs a two step process.
A badly written Extend client can perform requests which could take out the proxy server
*2. Use a cluster member if...*
<li> Your application runs for a long time - i.e. it is something like a web or application server
<li> Your application is well behaved - i.e. does not have big GC pauses. A very badly behaved cluster member can destabilize the rest of the cluster
<li> Your application is located close (in network terms) to the rest of the cluster
Pros:
Faster than an Extend client
Cons:
Needs to be stable
Needs to be located close to the rest of the cluster
Hope that helps as a start
JK -
Rolling Cluster Upgrade (OES2 to OES11) - GPT partition
We've got an OES2 SP3 cluster that we'll be rolling cluster upgrading to OES11 SP2.
We are currently at max capacity for several of the NSS volumes (2 TB).
If I'm reading/interpreting the OES11 SP2 docs correctly:
AFTER the cluster upgrade is complete, the only way I can get to larger volumes will be to create new LUNs on the SAN and intialize those with GPT. Then I could do the NSS Pool Move feature to get the existing 2 TB volumes to the larger setup?
Is that correct?
Or is there a better way that doesnt' require massive downtime?Originally Posted by konecnya
In article <[email protected]>, Kjhurni wrote:
> We are currently at max capacity for several of the NSS volumes (2 TB).
I was on the understanding that you could bind multiple partitions to
create NSS volumes up to 8TB. But I'd be hesitant to do that for a
clustered volume as well.
> I could do the NSS Pool Move feature to get the existing 2 TB
> volumes to the larger setup?
> Or is there a better way that doesnt' require massive downtime?
My first thought is the migration wizard so that the bulk of the copy can
be done while system is live but in the quieter times. Then the final
update with down time should be much faster.
But then do you really need down time for the Pool Move feature?
https://www.novell.com/documentation...a/bwtebhm.html
Certainly indicates it can be done live on a cluster with the move
process being cluster aware.
Andy of
http://KonecnyConsulting.ca in Toronto
Knowledge Partner
http://forums.novell.com/member.php/75037-konecnya
If you find a post helpful and are logged in the Web interface, please
show your appreciation by clicking on the star below. Thanks!
Migration = pew pew (re-doing IP's/etc and copying 4 TB of data is ugly especially when it's lots of tiny stuff).
haha
Anyway, when I asked about a better way that doesn't require massive downtime what I meant was:
Is there a better way vs. Pool Move that doesn't require massive downtime (in other words, the "other way" having the massive downtime, not the Pool Move).
Choice A = Pool Move = No downtime
But let's say that's not a good option (for whatever reason) and someone says use Choice B. But Choice B ends up requiring downtime (like the data copy option).
I just didn't know if Pool Move required that you create the partition ahead of time (so you can choose GPT) or if it kinda did it all for you on the fly (I'll have to read up when I get to that point).
I'm not terribly keep on having multiple DOS partitions, although that technically would work. Just always scares me. It's just temporary for the next 8 months anyway while we migrate to NAS from OES, but I'm running out of space and am on an unsupported OS anyway. -
[asr9k cluster upgrade procedure]
Dear CSC (and hopefully Xander):
What is the proper way of upgrading an asr9k cluster?
Do i have to break the cluster and upgrade both 9ks separately? then rebuild the cluster?
Or you just treat the cluster as one box and when you upgrade one of them, both are upgraded simultaneously?
(is there a document that describes this procedure for a cluster specifically?)
Thanks in advance!
c.Hello Carlos,
you can proceed with the following Cisco's recommendations thanks to Lenin Pedu:
https://supportforums.cisco.com/docs/DOC-34114#13_Cluster_RackByRack_Upgrade_
HTH,
Michel. -
Testing Coherence Cluster and Servers after WebLogic Console Creation
Hello,
I have created WLST scripts that extend a Domain with Coherence Clusters and Servers using unicast configurations. I can start and run the Coherence Servers from WL Admin Console without errors and warnings.WL 10.3.6
I am looking to test the configuration with something like coherence.sh and query.sh but I am missing instructions on how to use these tools with unicast and connect to the caches?
Is there a command line interface that connects to a Coherence Server cache created form the WL Admin Console using unicast? Do I need to override the any xml configuration to make this work?
Examples would be helpful.
While testing I have found the following....
I have changed coherence.sh and enabled storage. In addition:
JAVA_OPTS="-Xms$MEMORY -Xmx$MEMORY -Dtangosol.coherence.distributed.localstorage=$STORAGE_ENABLED $JMXPROPERTIES -Dtangosol.coherence.clusterport=7777 -Dtangosol.coherence.clusteraddress=231.1.1.1"
The Coherence Cluster configurations were changed to match the multicast settings for port and address above.
When this was performed all worked!!
However, if I changed JAVA_OPTS to use unicast
JAVA_OPTS="-Xms$MEMORY -Xmx$MEMORY -Dtangosol.coherence.distributed.localstorage=$STORAGE_ENABLED $JMXPROPERTIES -Dtangosol.coherence.localport=8088 -Dtangosol.coherence.localhost=192.168.2.69"
It fails to connect with the Coherence Server in the cluster.Hi there,
1. How did you achieve the https configuration in Weblogic ? And for which server Admin server or managed Server ?
2. Are you using which java key store ? Able to see the successful entries in <server>.out log file which is used for start up and stop of the weblogic server ?
Thanks
Laksh -
Issue to setup local Coherence cluster with WKA (well-known-address)
Hello - I have started local coherence cluster using WKA with single node,but when I start CacheFactory (coherence.cmd) with same configuration it throws following error message.
Any help is appricicated.
JVM startup Arrgument
-Dtangosol.coherence.override=cluster.xml
cluster.xml
<?xml version="1.0" encoding="UTF-8"?>
<coherence xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config http://xmlns.oracle.com/coherence/coherence-operational-config/1.1/coherence-operational-config.xsd">
<cluster-config>
<unicast-listener>
<well-known-addresses>
<socket-address id="1">
<address>171.193.103.25</address>
<port>8088</port>
</socket-address>
</well-known-addresses>
</unicast-listener>
</cluster-config>
<logging-config>
<destination>stdout</destination>
<severity-level>9</severity-level>
</logging-config>
</coherence>
Cluster startup Message
WellKnownAddressList(Size=1,
WKA{Address=171.193.103.25, Port=8088}
MasterMemberSet(
ThisMember=Member(Id=1, Timestamp=2013-10-24 11:07:18.603, Address=171.193.103.25:8088, MachineId=9041, Location=site:,machine:FD4C9EF534D5D,process:16704, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2013-10-24 11:07:18.603, Address=171.193.103.25:8088, MachineId=9041, Location=site:,machine:FD4C9EF534D5D,process:16704, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1
Member(Id=1, Timestamp=2013-10-24 11:07:18.603, Address=171.193.103.25:8088, MachineId=9041, Location=site:,machine:FD4C9EF534D5D,process:16704, Role=CoherenceServer)
MemberId|ServiceVersion|ServiceJoined|MemberState
1|3.7.1|2013-10-24 11:07:48.843|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}
2013-10-24 11:07:48.869/31.794 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
2013-10-24 11:07:49.058/31.983 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 1
2013-10-24 11:07:49.077/32.002 Oracle Coherence GE 3.7.1.0 <D6> (thread=DistributedCache, member=1): Service DistributedCache: sending PartitionConfig ConfigSync to all
2013-10-24 11:07:49.121/32.046 Oracle Coherence GE 3.7.1.0 <D5> (thread=ReplicatedCache, member=1): Service ReplicatedCache joined the cluster with senior service member 1
2013-10-24 11:07:49.128/32.053 Oracle Coherence GE 3.7.1.0 <D5> (thread=OptimisticCache, member=1): Service OptimisticCache joined the cluster with senior service member 1
2013-10-24 11:07:49.131/32.056 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocation:InvocationService, member=1): Service InvocationService joined the cluster with senior service member 1
2013-10-24 11:07:49.132/32.057 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=1):
Services
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
PartitionedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
ReplicatedCache{Name=ReplicatedCache, State=(SERVICE_STARTED), Id=3, Version=3.0, OldestMemberId=1}
Optimistic{Name=OptimisticCache, State=(SERVICE_STARTED), Id=4, Version=3.0, OldestMemberId=1}
InvocationService{Name=InvocationService, State=(SERVICE_STARTED), Id=5, Version=3.1, OldestMemberId=1}
Started DefaultCacheServer...
Error Message from CacheFactory
C:\Users\Zk5rjg8>C:\coherence37\bin\coherence.cmd
** Starting storage disabled console **
java version "1.6.0_51"
Java(TM) SE Runtime Environment (build 1.6.0_51-b11)
Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01, mixed mode)
2013-10-24 11:13:22.851/0.392 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/C:/coherence37/lib/coherence.jar!/tangosol-coherence.xml"
2013-10-24 11:13:22.920/0.462 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "file:/C:/coherence37/cluster.xml"
2013-10-24 11:13:22.924/0.465 Oracle Coherence 3.7.1.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
2013-10-24 11:13:22.924/0.465 Oracle Coherence 3.7.1.0 <D6> (thread=main, member=n/a): Loaded edition data from "jar:file:/C:/coherence37/lib/coherence.jar!/coherence-grid.xml"
Oracle Coherence Version 3.7.1.0 Build 27797
Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
2013-10-24 11:13:23.722/1.263 Oracle Coherence GE 3.7.1.0 <D4> (thread=main, member=n/a): TCMP bound to /171.193.103.25:8090 using SystemSocketProvider
2013-10-24 11:13:54.001/31.542 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2013-10-24 11:13:23.762, Address=171.193.103.25:8090, MachineId=9041, Location=site:,machine:FD4C9EF534D5D,process:17192, Role=CoherenceConsole) has been attempting to joi
2013-10-24 11:13:54.001/31.542 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:14:24.402/61.943 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:14:54.805/92.346 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:15:25.207/122.748 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:15:55.610/153.151 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:16:26.012/183.553 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:16:56.414/213.955 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:17:26.817/244.358 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:17:57.219/274.760 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:17:58.271/275.812 Oracle Coherence GE 3.7.1.0 <Error> (thread=Cluster, member=n/a): Detected soft timeout) of {WrapperGuardable Guard{Daemon=IpMonitor} Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_ANNOUNCE), Id=0, Version=3.7.1}}
2013-10-24 11:17:58.273/275.814 Oracle Coherence GE 3.7.1.0 <Error> (thread=Recovery Thread, member=n/a): Full Thread Dump
Thread[PacketListener1,8,Cluster]
java.net.PlainDatagramSocketImpl.receive0(Native Method)
java.net.PlainDatagramSocketImpl.receive(Unknown Source)
java.net.DatagramSocket.receive(Unknown Source)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Unknown Source)
Thread[PacketReceiver,7,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onWait(PacketReceiver.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Unknown Source)
Thread[Attach Listener,5,system]
Thread[PacketPublisher,6,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onWait(PacketPublisher.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Unknown Source)
Thread[Cluster|STATE_ANNOUNCE|Member(Id=0, Timestamp=2013-10-24 11:13:23.762, Address=171.193.103.25:8090, MachineId=9041, Location=site:,machine:FD4C9EF534D5D,process:17192, Role=CoherenceConsole),5,Cluster]
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
sun.nio.ch.SelectorImpl.select(Unknown Source)
com.tangosol.coherence.component.net.TcpRing.select(TcpRing.CDB:11)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onWait(ClusterService.CDB:6)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Unknown Source)
Thread[Reference Handler,10,system]
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Thread[Finalizer,8,system]
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(Unknown Source)
java.lang.ref.ReferenceQueue.remove(Unknown Source)
java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Thread[Signal Dispatcher,9,system]
Thread[PacketSpeaker,8,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketSpeaker.onNotify(PacketSpeaker.CDB:21)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Unknown Source)
Thread[Logger@1457155060 3.7.1.0,3,main]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Unknown Source)
Thread[PacketListener1P,8,Cluster]
java.net.PlainDatagramSocketImpl.receive0(Native Method)
java.net.PlainDatagramSocketImpl.receive(Unknown Source)
java.net.DatagramSocket.receive(Unknown Source)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Unknown Source)
Thread[main,5,main]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
com.tangosol.coherence.component.application.console.Coherence.run(Coherence.CDB:25)
com.tangosol.coherence.component.application.console.Coherence.main(Coherence.CDB:3)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
com.tangosol.net.CacheFactory.main(CacheFactory.java:827)
Thread[Recovery Thread,5,Cluster]
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Unknown Source)
com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:810)
com.tangosol.internal.net.cluster.DefaultServiceFailurePolicy.onGuardableRecovery(DefaultServiceFailurePolicy.java:44)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$WrapperGuardable.recover(Grid.CDB:1)
com.tangosol.net.GuardSupport$Context$1.run(GuardSupport.java:653)
java.lang.Thread.run(Unknown Source)
2013-10-24 11:17:58.273/275.814 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Recovery Thread, member=n/a): Attempting recovery of Guard{Daemon=IpMonitor}
Exception in thread "main" 2013-10-24 11:18:24.025/301.566 Oracle Coherence GE 3.7.1.0 <Error> (thread=main, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.tangosol.coherence.component.application.console.Coherence.run(Coherence.CDB:25)
at com.tangosol.coherence.component.application.console.Coherence.main(Coherence.CDB:3)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.tangosol.net.CacheFactory.main(CacheFactory.java:827)
java.lang.reflect.InvocationTargetException
2013-10-24 11:18:24.025/301.566 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Service Cluster left the cluster at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.tangosol.net.CacheFactory.main(CacheFactory.java:827)
Caused by: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.tangosol.coherence.component.application.console.Coherence.run(Coherence.CDB:25)
at com.tangosol.coherence.component.application.console.Coherence.main(Coherence.CDB:3)
... 5 more
C:\Users\Zk5rjg8>Hi SajeevPynadath
1
First start the server process with "cache-server.cmd"
2
After that you can start another server or client process, the "coherence.cmd" script is to start a client process to join the cluster .
3
Then now you have 2 processes , and your cluster.xml will look like this :
<socket-address id="serverprocess">
<address>171.193.103.25</address>
<port>8088</port>
</socket-address>
<socket-address id="clienprocess">
<address>171.193.103.25</address>
<port>8089</port>
</socket-address>
4
Before start each process remember put in java command line :
for server
-Dtangosol.coherence.localhost=171.193.103.25 -Dtangosol.coherence.localport=8088
for client
-Dtangosol.coherence.localhost=171.193.103.25 -Dtangosol.coherence.localport=8089
regards,
Leo_TA -
Could anyone tell me what is a senior member in a coherence cluster?
Hi all,
I am problem finding the definition of Senior member in coherence documentation.
Could anyone tell me what is a Senior member in a coherence cluster?
Could we configure which server is Senior? How?
Thanks and Regards,
NBHi NB,
a senior member of a clustered service (there is one for each service) is the member which has special role in deciding service membership related issues. There is always one (usually the oldest living member), if it dies, then a new one is immediately elected.
Since the cluster is also a service it also has a senior member, and that is referred to as the senior member of the cluster.
You cannot configure which node is the senior for the cluster (or for any service for that matter), that is determined by the timing of the nodes being started, the senior will usually be the node started first unless they are started very quickly during cluster formation, at which point it can seem random from the outside.
Some parts of the documentation hints that the distributed cache services have a storage-senior node, too, which is not exactly the same thing as a service senior, as you can start a distributed cache service in a way that storage-disabled members happened to start first in which case the senior member for the service would not be storage-enabled and there would be no storage senior since there are no storage nodes until the first is started. You can consider this as a service (storage functionality) within a service (distributed cache service), similarly to how clustered services relate to the ClusterService.
Best regards,
Robert -
Any samples for using JMX for displaying cluster statistics 3.6 ?
Hi,
Our Coherence 3.6 cluster is configured in the WLS domain and controlled using WLS node manager. This is all working well. We have created a WLS console extension to show the actual live cluster topology (rather than the configured topology) so we can ensure all the domains are configured correctly and there are no nodes in the cluster that there should not be (we have many domains/environments).
I can show basic cluster statistics and show the members etc using the CacheFactory. But for each cluster member I would like to show if it is storage enabled and free memory and other statistics. I was expecting to get that from the Member or Cluster objects returned from the CacheFactory methods but it seems not. It seems that JMX is the way to go. Does anyone have any sample JMX code they might share that would connect to the cluster and retrieve stats and info that is not available from the CacheFactory api ?.
TIA
MartinHi Revanth,
The issue seems to be with the JMX URL value defined in the 'server URL' property.
Apparently I ran into the same issue as I was unable to access the Admin Server URL from the SMA.
Changing the Host and Port value properly resolved the issue for me.
Regards,
Anandraj -
"cluster.ClusteredHandlerFactory: Failed to initialize Coherence Cluster"
Hi,
I have a cluster with Admin, Proxy and MS1 on one server and MS2 ,MS3 on another server.
Below is the content of tangosol-coherence.xml file modified according to my env set up (The file is taken from Middleware1036/coherence_3.7/lib/coherence.jar and retained the content that is only required for the unicast configuration)
<cluster-config>
<member-identity>
<!--
Choose a unique name for the cluster, such that no other OSM instance in your
network would have the same name
-->
<cluster-name system-property="tangosol.coherence.cluster">ThirdCluster
</cluster-name>
</member-identity>
<unicast-listener>
<!--
Add more <socket-address>socket_address</socket-address> elements to well-known
addresses, one for each member in your OSM WebLogic cluster.
-->
<well-known-addresses>
<socket-address id="1">
<address>host1.example.com</address>
<port>31171</port>
</socket-address>
<socket-address id="2">
<address>host2.example.com</address>
<port>31172</port>
</socket-address>
<socket-address id="3">
<address>host2.example.com</address>
<port>31173</port>
</socket-address>
</well-known-addresses>
</unicast-listener>
</cluster-config>
After configuring the cluster domain with coherence, I made all the managed servers up with Admin and proxy and installed my application(Oracle communications order and service management product) into the cluster.
As per recommendation I need to restart all the servers to see my application osm.ear in active state.
Admin, proxy got restarted but while restarting the managed servers below is the error message I am encountering:
Hence please assist me on why is it saying "Failed to initialize the coherence cluster" .
####<Aug 6, 2012 6:20:04 AM PDT> <Error> <oms> <blr2230328> <ms1> <[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <oms-internal> <> <0000JZw4SMKB1FHpMs8Dye1G7wBy000001> <1344259204521> <BEA-000000> <cluster.ClusteredHandlerFactory: Failed to initialize Coherence cluster
com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.mslv.oms.handler.cluster.g.refresh(Unknown Source)
at oracle.communications.ordermanagement.listener.impl.a.a(Unknown Source)
at com.mslv.oms.handler.cluster.ClusteredHandlerFactory.<clinit>(Unknown Source)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at com.mslv.oms.security.HandlerFactory.b(Unknown Source)
at com.mslv.oms.security.HandlerFactory.startup(Unknown Source)
at com.mslv.oms.j2ee.LifecycleListener.postStart(Unknown Source)
at weblogic.application.internal.flow.BaseLifecycleFlow$PostStartAction.run(BaseLifecycleFlow.java:297)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.application.internal.flow.BaseLifecycleFlow$LifecycleListenerAction.invoke(BaseLifecycleFlow.java:199)
at weblogic.application.internal.flow.BaseLifecycleFlow.postStart(BaseLifecycleFlow.java:71)
at weblogic.application.internal.flow.TailLifecycleFlow.activate(TailLifecycleFlow.java:33)
at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:671)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:212)
at weblogic.application.internal.EarDeployment.activate(EarDeployment.java:59)
at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:161)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79)
at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicDeployment.java:184)
at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromServerLifecycle(BasicDeployment.java:361)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doActivate(DeploymentAdapter.java:51)
at weblogic.management.deploy.internal.DeploymentAdapter.activate(DeploymentAdapter.java:200)
at weblogic.management.deploy.internal.AppTransition$2.transitionApp(AppTransition.java:30)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
at weblogic.management.deploy.internal.ConfiguredDeployments.activate(ConfiguredDeployments.java:169)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:123)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:180)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:96)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)user768552 wrote:
Hi,
I have a cluster with Admin, Proxy and MS1 on one server and MS2 ,MS3 on another server.
Below is the content of tangosol-coherence.xml file modified according to my env set up (The file is taken from Middleware1036/coherence_3.7/lib/coherence.jar and retained the content that is only required for the unicast configuration)
<cluster-config>
<member-identity>
<!--
Choose a unique name for the cluster, such that no other OSM instance in your
network would have the same name
-->
<cluster-name system-property="tangosol.coherence.cluster">ThirdCluster
</cluster-name>
</member-identity>
<unicast-listener>
<!--
Add more <socket-address>socket_address</socket-address> elements to well-known
addresses, one for each member in your OSM WebLogic cluster.
-->
<well-known-addresses>
<socket-address id="1">
<address>host1.example.com</address>
<port>31171</port>
</socket-address>
<socket-address id="2">
<address>host2.example.com</address>
<port>31172</port>
</socket-address>
<socket-address id="3">
<address>host2.example.com</address>
<port>31173</port>
</socket-address>
</well-known-addresses>
</unicast-listener>
</cluster-config>
After configuring the cluster domain with coherence, I made all the managed servers up with Admin and proxy and installed my application(Oracle communications order and service management product) into the cluster.
As per recommendation I need to restart all the servers to see my application osm.ear in active state.
Admin, proxy got restarted but while restarting the managed servers below is the error message I am encountering:
Hence please assist me on why is it saying "Failed to initialize the coherence cluster" .
####<Aug 6, 2012 6:20:04 AM PDT> <Error> <oms> <blr2230328> <ms1> <[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <oms-internal> <> <0000JZw4SMKB1FHpMs8Dye1G7wBy000001> <1344259204521> <BEA-000000> <cluster.ClusteredHandlerFactory: Failed to initialize Coherence cluster
com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.mslv.oms.handler.cluster.g.refresh(Unknown Source)
at oracle.communications.ordermanagement.listener.impl.a.a(Unknown Source)
at com.mslv.oms.handler.cluster.ClusteredHandlerFactory.<clinit>(Unknown Source)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at com.mslv.oms.security.HandlerFactory.b(Unknown Source)
at com.mslv.oms.security.HandlerFactory.startup(Unknown Source)
at com.mslv.oms.j2ee.LifecycleListener.postStart(Unknown Source)
at weblogic.application.internal.flow.BaseLifecycleFlow$PostStartAction.run(BaseLifecycleFlow.java:297)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.application.internal.flow.BaseLifecycleFlow$LifecycleListenerAction.invoke(BaseLifecycleFlow.java:199)
at weblogic.application.internal.flow.BaseLifecycleFlow.postStart(BaseLifecycleFlow.java:71)
at weblogic.application.internal.flow.TailLifecycleFlow.activate(TailLifecycleFlow.java:33)
at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:671)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:212)
at weblogic.application.internal.EarDeployment.activate(EarDeployment.java:59)
at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:161)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79)
at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicDeployment.java:184)
at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromServerLifecycle(BasicDeployment.java:361)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doActivate(DeploymentAdapter.java:51)
at weblogic.management.deploy.internal.DeploymentAdapter.activate(DeploymentAdapter.java:200)
at weblogic.management.deploy.internal.AppTransition$2.transitionApp(AppTransition.java:30)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
at weblogic.management.deploy.internal.ConfiguredDeployments.activate(ConfiguredDeployments.java:169)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:123)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:180)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:96)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)Seems like, the OSM application is unable to join the cluster. Modify the WKA definition as below:
<well-known-addresses>
<socket-address id="1">
<address system-property="tangosol.coherence.wka1"></address>
<port system-property="tangosol.coherence.wka1.port"></port>
</socket-address>
</well-known-addresses>
Now, in the ServerStart of the ManagedServers running the OSM application add the following -Dtangosol.coherence.wka1=<> -Dtangosol.coherence.wka1.port=<>
Hope this helps!
Cheers,
NJ -
The coherence cluster supports created how much cache?
Hi,
I have runing Oracle Coherence GE 3.6.1 in cluster.
The coherence cluster supports created how much cache? cache type is Distributed Cache.
Cache over-population will affect the performance of the cluster?
Thank you!Do a test, the code is as follows：
cache code
public final long removeLike(String regex) {
long l1 = System.currentTimeMillis();
long result = 0;
Set<String> setKeys = nc.keySet(new LikeFilter(new KeyExtractor(), regex + "%", (char)0, false));
long l2 = System.currentTimeMillis();
System.out.println("removeLike , execute time :" + (l2 - l1));
return result;
public final boolean removeStartWith(String prefix) {
long l1 = System.currentTimeMillis();
int i = 0;
try {
Set<String> keys = nc.keySet();
Iterator<String> iter = keys.iterator();
String v = "";
while (iter.hasNext()) {
v = iter.next();
if (v.startsWith(prefix)) {
++i;
long l2 = System.currentTimeMillis();
System.out.println("removeStartWith , execute time :" + (l2 - l1));
return true;
} catch (Exception e) {
LOG.error(e.getMessage(), e);
return false;
The above 2 methods are computational search KEY time, not to delete operation
test code
@BeforeClass
public static void init() {
try {
cache = CacheFactory.getCache("testCache");
} catch (CacheException e) {
e.printStackTrace();
cache.clear();
for (int i = 0; i < 100000; i++) {
cache.add("test-" + i,i);
for (int i = 0; i < 100000; i++) {
cache.add("hello-" + i,i);
@Test
public void testReadByDefaultKey1() {
cache.removeLike("test");
@Test
public void testReadByDefaultKey2() {
cache.removeStartWith("test");
The test results show that the:
removeLike , execute time : 637
removeStartWith , execute time :125
Circulating KEY faster than using likeFilter quickly.
Maybe you are looking for
-
Hi, Am getting very strange error while running a report. It gives me an 'Autopass' error while inserting some variables in the filter screen. Also it shows "system error in program CL RSR WWW RENDERER and form GET VARDEF FOR F4:03. Kindly provide yo
-
Text boxes not appearing - CAPTIVATE1.1
I have a number of delegates using Cap1.1 - on some machines when recording screens, the automatic text boxes are not appearing when the recording has finished. (the "text box" option is checked on all PCs in the custom options settings) I`m sure i`v
-
Pictures not appearing in iphoto library
I'm using version 2.0.1 of iphoto. Some of the pictures that are in my picture folder will not appear in the photo library. I have "rebuilt" iphoto, but some of the more recent pictures do not appear in the library. Any ideas on what else I could do
-
Why, after opening a PDF file, does the Acrobat Pro XI sit there, not reacting to any inputs until about a full minute goes by?
-
When trying to login in OIM 11g after providing credentials, it gives internal error 500.
When trying to login in OIM 11g after providing credentials, it gives internal error 500.