Issue to setup local Coherence cluster with WKA (well-known-address)
Hello - I have started local coherence cluster using WKA with single node,but when I start CacheFactory (coherence.cmd) with same configuration it throws following error message.
Any help is appricicated.
JVM startup Arrgument
-Dtangosol.coherence.override=cluster.xml
cluster.xml
<?xml version="1.0" encoding="UTF-8"?>
<coherence xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config http://xmlns.oracle.com/coherence/coherence-operational-config/1.1/coherence-operational-config.xsd">
<cluster-config>
<unicast-listener>
<well-known-addresses>
<socket-address id="1">
<address>171.193.103.25</address>
<port>8088</port>
</socket-address>
</well-known-addresses>
</unicast-listener>
</cluster-config>
<logging-config>
<destination>stdout</destination>
<severity-level>9</severity-level>
</logging-config>
</coherence>
Cluster startup Message
WellKnownAddressList(Size=1,
WKA{Address=171.193.103.25, Port=8088}
MasterMemberSet(
ThisMember=Member(Id=1, Timestamp=2013-10-24 11:07:18.603, Address=171.193.103.25:8088, MachineId=9041, Location=site:,machine:FD4C9EF534D5D,process:16704, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2013-10-24 11:07:18.603, Address=171.193.103.25:8088, MachineId=9041, Location=site:,machine:FD4C9EF534D5D,process:16704, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1
Member(Id=1, Timestamp=2013-10-24 11:07:18.603, Address=171.193.103.25:8088, MachineId=9041, Location=site:,machine:FD4C9EF534D5D,process:16704, Role=CoherenceServer)
MemberId|ServiceVersion|ServiceJoined|MemberState
1|3.7.1|2013-10-24 11:07:48.843|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}
2013-10-24 11:07:48.869/31.794 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
2013-10-24 11:07:49.058/31.983 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 1
2013-10-24 11:07:49.077/32.002 Oracle Coherence GE 3.7.1.0 <D6> (thread=DistributedCache, member=1): Service DistributedCache: sending PartitionConfig ConfigSync to all
2013-10-24 11:07:49.121/32.046 Oracle Coherence GE 3.7.1.0 <D5> (thread=ReplicatedCache, member=1): Service ReplicatedCache joined the cluster with senior service member 1
2013-10-24 11:07:49.128/32.053 Oracle Coherence GE 3.7.1.0 <D5> (thread=OptimisticCache, member=1): Service OptimisticCache joined the cluster with senior service member 1
2013-10-24 11:07:49.131/32.056 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocation:InvocationService, member=1): Service InvocationService joined the cluster with senior service member 1
2013-10-24 11:07:49.132/32.057 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=1):
Services
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
PartitionedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
ReplicatedCache{Name=ReplicatedCache, State=(SERVICE_STARTED), Id=3, Version=3.0, OldestMemberId=1}
Optimistic{Name=OptimisticCache, State=(SERVICE_STARTED), Id=4, Version=3.0, OldestMemberId=1}
InvocationService{Name=InvocationService, State=(SERVICE_STARTED), Id=5, Version=3.1, OldestMemberId=1}
Started DefaultCacheServer...
Error Message from CacheFactory
C:\Users\Zk5rjg8>C:\coherence37\bin\coherence.cmd
** Starting storage disabled console **
java version "1.6.0_51"
Java(TM) SE Runtime Environment (build 1.6.0_51-b11)
Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01, mixed mode)
2013-10-24 11:13:22.851/0.392 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/C:/coherence37/lib/coherence.jar!/tangosol-coherence.xml"
2013-10-24 11:13:22.920/0.462 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "file:/C:/coherence37/cluster.xml"
2013-10-24 11:13:22.924/0.465 Oracle Coherence 3.7.1.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
2013-10-24 11:13:22.924/0.465 Oracle Coherence 3.7.1.0 <D6> (thread=main, member=n/a): Loaded edition data from "jar:file:/C:/coherence37/lib/coherence.jar!/coherence-grid.xml"
Oracle Coherence Version 3.7.1.0 Build 27797
Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
2013-10-24 11:13:23.722/1.263 Oracle Coherence GE 3.7.1.0 <D4> (thread=main, member=n/a): TCMP bound to /171.193.103.25:8090 using SystemSocketProvider
2013-10-24 11:13:54.001/31.542 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2013-10-24 11:13:23.762, Address=171.193.103.25:8090, MachineId=9041, Location=site:,machine:FD4C9EF534D5D,process:17192, Role=CoherenceConsole) has been attempting to joi
2013-10-24 11:13:54.001/31.542 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:14:24.402/61.943 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:14:54.805/92.346 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:15:25.207/122.748 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:15:55.610/153.151 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:16:26.012/183.553 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:16:56.414/213.955 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:17:26.817/244.358 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:17:57.219/274.760 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond
2013-10-24 11:17:58.271/275.812 Oracle Coherence GE 3.7.1.0 <Error> (thread=Cluster, member=n/a): Detected soft timeout) of {WrapperGuardable Guard{Daemon=IpMonitor} Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_ANNOUNCE), Id=0, Version=3.7.1}}
2013-10-24 11:17:58.273/275.814 Oracle Coherence GE 3.7.1.0 <Error> (thread=Recovery Thread, member=n/a): Full Thread Dump
Thread[PacketListener1,8,Cluster]
java.net.PlainDatagramSocketImpl.receive0(Native Method)
java.net.PlainDatagramSocketImpl.receive(Unknown Source)
java.net.DatagramSocket.receive(Unknown Source)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Unknown Source)
Thread[PacketReceiver,7,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onWait(PacketReceiver.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Unknown Source)
Thread[Attach Listener,5,system]
Thread[PacketPublisher,6,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onWait(PacketPublisher.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Unknown Source)
Thread[Cluster|STATE_ANNOUNCE|Member(Id=0, Timestamp=2013-10-24 11:13:23.762, Address=171.193.103.25:8090, MachineId=9041, Location=site:,machine:FD4C9EF534D5D,process:17192, Role=CoherenceConsole),5,Cluster]
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
sun.nio.ch.SelectorImpl.select(Unknown Source)
com.tangosol.coherence.component.net.TcpRing.select(TcpRing.CDB:11)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onWait(ClusterService.CDB:6)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Unknown Source)
Thread[Reference Handler,10,system]
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Thread[Finalizer,8,system]
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(Unknown Source)
java.lang.ref.ReferenceQueue.remove(Unknown Source)
java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Thread[Signal Dispatcher,9,system]
Thread[PacketSpeaker,8,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketSpeaker.onNotify(PacketSpeaker.CDB:21)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Unknown Source)
Thread[Logger@1457155060 3.7.1.0,3,main]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Unknown Source)
Thread[PacketListener1P,8,Cluster]
java.net.PlainDatagramSocketImpl.receive0(Native Method)
java.net.PlainDatagramSocketImpl.receive(Unknown Source)
java.net.DatagramSocket.receive(Unknown Source)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Unknown Source)
Thread[main,5,main]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
com.tangosol.coherence.component.application.console.Coherence.run(Coherence.CDB:25)
com.tangosol.coherence.component.application.console.Coherence.main(Coherence.CDB:3)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
com.tangosol.net.CacheFactory.main(CacheFactory.java:827)
Thread[Recovery Thread,5,Cluster]
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Unknown Source)
com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:810)
com.tangosol.internal.net.cluster.DefaultServiceFailurePolicy.onGuardableRecovery(DefaultServiceFailurePolicy.java:44)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$WrapperGuardable.recover(Grid.CDB:1)
com.tangosol.net.GuardSupport$Context$1.run(GuardSupport.java:653)
java.lang.Thread.run(Unknown Source)
2013-10-24 11:17:58.273/275.814 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Recovery Thread, member=n/a): Attempting recovery of Guard{Daemon=IpMonitor}
Exception in thread "main" 2013-10-24 11:18:24.025/301.566 Oracle Coherence GE 3.7.1.0 <Error> (thread=main, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.tangosol.coherence.component.application.console.Coherence.run(Coherence.CDB:25)
at com.tangosol.coherence.component.application.console.Coherence.main(Coherence.CDB:3)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.tangosol.net.CacheFactory.main(CacheFactory.java:827)
java.lang.reflect.InvocationTargetException
2013-10-24 11:18:24.025/301.566 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Service Cluster left the cluster at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.tangosol.net.CacheFactory.main(CacheFactory.java:827)
Caused by: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.tangosol.coherence.component.application.console.Coherence.run(Coherence.CDB:25)
at com.tangosol.coherence.component.application.console.Coherence.main(Coherence.CDB:3)
... 5 more
C:\Users\Zk5rjg8>
Hi SajeevPynadath
1
First start the server process with "cache-server.cmd"
2
After that you can start another server or client process, the "coherence.cmd" script is to start a client process to join the cluster .
3
Then now you have 2 processes , and your cluster.xml will look like this :
<socket-address id="serverprocess">
<address>171.193.103.25</address>
<port>8088</port>
</socket-address>
<socket-address id="clienprocess">
<address>171.193.103.25</address>
<port>8089</port>
</socket-address>
4
Before start each process remember put in java command line :
for server
-Dtangosol.coherence.localhost=171.193.103.25 -Dtangosol.coherence.localport=8088
for client
-Dtangosol.coherence.localhost=171.193.103.25 -Dtangosol.coherence.localport=8089
regards,
Leo_TA
Similar Messages
-
Well Known Address (wka) issue
Hi,
I am hitting some issues whilst trying to configure and use multiple wkas in my cluster using Coherence 3.5.3.
It appears that only the first WKA is being picked up from the configuration file, rather than the full list.
For example, if I include the following in my tangosol-coherence-override.xml config file:
<well-known-addresses>
<socket-address id="1">
<address>cyclone</address>
<port system-property="tangosol.coherence.wka.port"></port>
</socket-address>
<socket-address id="2">
<address>tornado</address>
<port system-property="tangosol.coherence.wka.port"></port>
</socket-address>
</well-known-addresses>
The logs will indicate that only a single wka is available (i.e. cyclone).
If we try and bring proxy and storage nodes up on tornado they will not start up, as they will attempt to use cyclone.
Does anyone have any ideas about where we might be going wrong?
CheersHi Luk,
Below is the tangosol-coherence-override.xml file I am using:
Many Thanks
<?xml version='1.0'?>
<!-- Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademarks of Oracle Corporation and/or its affiliates.
This software is the confidential and proprietary information of Oracle Corporation.
You shall not disclose such confidential and proprietary information and
shall use it only in accordance with the terms of the license agreement you
entered into with Oracle. This notice may not be removed or altered. -->
<!DOCTYPE coherence SYSTEM "coherence.dtd">
<!-- Note: This XML is set up for a development mode; search for "Note:"
to find specific notes on changes suggested for production use. -->
<coherence>
<cluster-config>
<member-identity>
<!-- Note: All members of the cluster must specify the same cluster name
in order to be allowed to join the cluster. -->
<cluster-name system-property="tangosol.coherence.cluster"></cluster-name>
<site-name system-property="tangosol.coherence.site"></site-name>
<rack-name system-property="tangosol.coherence.rack"></rack-name>
<machine-name system-property="tangosol.coherence.machine"></machine-name>
<process-name system-property="tangosol.coherence.process"></process-name>
<member-name system-property="tangosol.coherence.member"></member-name>
<role-name system-property="tangosol.coherence.role"></role-name>
<priority system-property="tangosol.coherence.priority"></priority>
</member-identity>
<unicast-listener>
<!-- Note: Specifying one or more "well known" addresses (WKA) will disable
all multicast communication. In this case, for a member to join the cluster
it will either have to be a WKA or there will have to be at least one WKA
member running. -->
<well-known-addresses>
<socket-address id="1">
<address>cyclone</address>
<port system-property="tangosol.coherence.wka.port"></port>
</socket-address>
<socket-address id="2">
<address>tornado</address>
<port system-property="tangosol.coherence.wka.port"></port>
</socket-address>
</well-known-addresses>
<!-- Note: The "localhost" setting may not work on systems that define
localhost as the loopback address; in that case, specify the machine name
or the specific IP address. -->
<address system-property="tangosol.coherence.localhost">localhost</address>
<port system-property="tangosol.coherence.localport">8088</port>
<port-auto-adjust system-property="tangosol.coherence.localport.adjust">true</port-auto-adjust>
<packet-buffer>
<maximum-packets>1428</maximum-packets>
</packet-buffer>
<priority>8</priority>
</unicast-listener>
<tcp-ring-listener>
<!-- Note: When encountering problems getting Coherence running on a system,
we suggest that this value be set to "false" until all other issues are resolved
in order to simplify trouble-shooting. -->
<enabled system-property="tangosol.coherence.tcpring">true</enabled>
<maximum-socket-closed-exceptions>2</maximum-socket-closed-exceptions>
<priority>6</priority>
</tcp-ring-listener>
<shutdown-listener>
<!-- Note: For production use, the suggested value is "none" (or "false")
unless testing has verified that the behavior on external shutdown is exactly
what is desired. -->
<enabled system-property="tangosol.coherence.shutdownhook">force</enabled>
</shutdown-listener>
<service-guardian>
<service-failure-policy>exit-cluster</service-failure-policy>
<!-- Note: For production use, the recommended value is equal to or greater
than the packet-delivery timeout. -->
<timeout-milliseconds system-property="tangosol.coherence.guard.timeout">65000</timeout-milliseconds>
</service-guardian>
<packet-speaker>
<priority>8</priority>
</packet-speaker>
<packet-publisher>
<packet-size>
<maximum-length>1468</maximum-length>
<preferred-length>1468</preferred-length>
</packet-size>
<packet-pool>
<maximum-packets>2048</maximum-packets>
</packet-pool>
<packet-delivery>
<resend-milliseconds>200</resend-milliseconds>
<!-- Note: For production use, the recommended value is the greater of
60000 and two times the maximum expected full GC duration. -->
<timeout-milliseconds>60000</timeout-milliseconds>
<heartbeat-milliseconds>1000</heartbeat-milliseconds>
<flow-control>
<enabled>true</enabled>
<pause-detection>
<maximum-packets>16</maximum-packets>
</pause-detection>
<outstanding-packets>
<maximum-packets>4096</maximum-packets>
<minimum-packets>64</minimum-packets>
</outstanding-packets>
</flow-control>
<packet-bundling>
<maximum-deferral-time>1us</maximum-deferral-time>
<aggression-factor>0</aggression-factor>
</packet-bundling>
</packet-delivery>
<notification-queueing>
<ack-delay-milliseconds>16</ack-delay-milliseconds>
<nack-delay-milliseconds>1</nack-delay-milliseconds>
</notification-queueing>
<burst-mode>
<maximum-packets>0</maximum-packets>
<pause-milliseconds>10</pause-milliseconds>
</burst-mode>
<traffic-jam>
<maximum-packets>8192</maximum-packets>
<pause-milliseconds>10</pause-milliseconds>
</traffic-jam>
<packet-buffer>
<maximum-packets>32</maximum-packets>
</packet-buffer>
<priority>6</priority>
<!-- Note: Disabling the Publisher is used only by Java Real-Time Clients
to restrict their ability to use the cluster protocol (TCMP). When disabled,
a client is only be able to access clustered services via Extend proxies. -->
<enabled system-property="tangosol.coherence.tcmp.enabled">true</enabled>
</packet-publisher>
<incoming-message-handler>
<maximum-time-variance>16</maximum-time-variance>
<packet-pool>
<maximum-packets>2048</maximum-packets>
</packet-pool>
<use-nack-packets>true</use-nack-packets>
<priority>7</priority>
</incoming-message-handler>
<outgoing-message-handler>
<!-- Note: Uncomment the following section to activate gzip compression
for all network messages, which can help substantially with WAN and low-bandwidth
networks; for more information, see: http://www.tangosol.com/faq-coherence.jsp#filterconfig -->
<use-filters>
<!-- <filter-name>gzip</filter-name> -->
</use-filters>
</outgoing-message-handler>
<authorized-hosts>
<host-address></host-address>
<host-range>
<from-address></from-address>
<to-address></to-address>
</host-range>
</authorized-hosts>
<services>
<service id="1">
<service-type>ReplicatedCache</service-type>
<service-component>ReplicatedCache</service-component>
<use-filters>
<filter-name />
</use-filters>
<init-params>
<init-param id="1">
<param-name>standard-lease-milliseconds</param-name>
<param-value>0</param-value>
</init-param>
<init-param id="2">
<param-name>lease-granularity</param-name>
<param-value>thread</param-value>
</init-param>
<init-param id="3">
<param-name>mobile-issues</param-name>
<param-value>false</param-value>
</init-param>
<init-param id="4">
<param-name>request-timeout</param-name>
<param-value system-property="tangosol.coherence.replicated.request.timeout"></param-value>
</init-param>
<init-param id="5">
<param-name>serializer</param-name>
<param-value></param-value>
</init-param>
<init-param id="6">
<param-name>guardian-timeout</param-name>
<param-value system-property="tangosol.coherence.replicated.guard.timeout"></param-value>
</init-param>
</init-params>
</service>
<service id="2">
<service-type>OptimisticCache</service-type>
<service-component>ReplicatedCache.Optimistic</service-component>
<use-filters>
<filter-name />
</use-filters>
<init-params>
<init-param id="1">
<param-name>request-timeout</param-name>
<param-value system-property="tangosol.coherence.optimistic.request.timeout"></param-value>
</init-param>
<init-param id="2">
<param-name>serializer</param-name>
<param-value></param-value>
</init-param>
<init-param id="3">
<param-name>guardian-timeout</param-name>
<param-value system-property="tangosol.coherence.optimistic.guard.timeout"></param-value>
</init-param>
</init-params>
</service>
<service id="3">
<service-type>DistributedCache</service-type>
<service-component>DistributedCache</service-component>
<use-filters>
<filter-name />
</use-filters>
<init-params>
<init-param id="1">
<param-name>standard-lease-milliseconds</param-name>
<param-value>0</param-value>
</init-param>
<init-param id="2">
<param-name>lease-granularity</param-name>
<param-value>thread</param-value>
</init-param>
<init-param id="3">
<param-name>partition-count</param-name>
<param-value>257</param-value>
</init-param>
<init-param id="4">
<param-name>local-storage</param-name>
<param-value system-property="tangosol.coherence.distributed.localstorage">true</param-value>
</init-param>
<init-param id="5">
<param-name>transfer-threshold</param-name>
<param-value system-property="tangosol.coherence.distributed.transfer">512</param-value>
</init-param>
<init-param id="6">
<param-name>backup-count</param-name>
<param-value system-property="tangosol.coherence.distributed.backupcount">1</param-value>
</init-param>
<init-param id="7">
<param-name>backup-storage/type</param-name>
<param-value system-property="tangosol.coherence.distributed.backup">on-heap</param-value>
</init-param>
<init-param id="8">
<param-name>thread-count</param-name>
<param-value system-property="tangosol.coherence.distributed.threads">5</param-value>
</init-param>
<init-param id="9">
<param-name>key-associator/class-name</param-name>
<param-value></param-value>
</init-param>
<init-param id="10">
<param-name>key-partitioning/class-name</param-name>
<param-value></param-value>
</init-param>
<init-param id="11">
<param-name>partition-listener/class-name</param-name>
<param-value></param-value>
</init-param>
<init-param id="12">
<param-name>task-hung-threshold</param-name>
<param-value system-property="tangosol.coherence.distributed.task.hung"></param-value>
</init-param>
<init-param id="13">
<param-name>task-timeout</param-name>
<param-value system-property="tangosol.coherence.distributed.task.timeout"></param-value>
</init-param>
<init-param id="14">
<param-name>request-timeout</param-name>
<param-value system-property="tangosol.coherence.distributed.request.timeout"></param-value>
</init-param>
<init-param id="15">
<param-name>serializer</param-name>
<param-value></param-value>
</init-param>
<init-param id="16">
<param-name>backup-count-after-writebehind</param-name>
<param-value></param-value>
</init-param>
<init-param id="17">
<param-name>guardian-timeout</param-name>
<param-value system-property="tangosol.coherence.distributed.guard.timeout"></param-value>
</init-param>
</init-params>
</service>
<service id="4">
<service-type>Invocation</service-type>
<service-component>InvocationService</service-component>
<use-filters>
<filter-name />
</use-filters>
<init-params>
<init-param id="1">
<param-name>thread-count</param-name>
<param-value system-property="tangosol.coherence.invocation.threads">5</param-value>
</init-param>
<init-param id="2">
<param-name>task-hung-threshold</param-name>
<param-value system-property="tangosol.coherence.invocation.task.hung"></param-value>
</init-param>
<init-param id="3">
<param-name>task-timeout</param-name>
<param-value system-property="tangosol.coherence.invocation.task.timeout"></param-value>
</init-param>
<init-param id="4">
<param-name>request-timeout</param-name>
<param-value system-property="tangosol.coherence.invocation.request.timeout"></param-value>
</init-param>
<init-param id="5">
<param-name>serializer</param-name>
<param-value></param-value>
</init-param>
<init-param id="6">
<param-name>guardian-timeout</param-name>
<param-value system-property="tangosol.coherence.invocation.guard.timeout"></param-value>
</init-param>
</init-params>
</service>
<service id="5">
<service-type>LocalCache</service-type>
<service-component>LocalCache</service-component>
<init-params>
<init-param id="1">
<param-name>lock-enforce</param-name>
<param-value>false</param-value>
</init-param>
<init-param id="2">
<param-name>lock-wait</param-name>
<param-value>0</param-value>
</init-param>
</init-params>
</service>
<service id="6">
<service-type>SimpleCache</service-type>
<service-component>SimpleCache</service-component>
</service>
<service id="7">
<service-type>Proxy</service-type>
<service-component>ProxyService</service-component>
<use-filters>
<filter-name />
</use-filters>
<init-params>
<init-param id="1">
<param-name>acceptor-config</param-name>
<param-value></param-value>
</init-param>
<init-param id="2">
<param-name>proxy-config</param-name>
<param-value></param-value>
</init-param>
<init-param id="3">
<param-name>thread-count</param-name>
<param-value system-property="tangosol.coherence.proxy.threads">10</param-value>
</init-param>
<init-param id="4">
<param-name>task-hung-threshold</param-name>
<param-value system-property="tangosol.coherence.proxy.task.hung"></param-value>
</init-param>
<init-param id="5">
<param-name>task-timeout</param-name>
<param-value system-property="tangosol.coherence.proxy.task.timeout"></param-value>
</init-param>
<init-param id="6">
<param-name>request-timeout</param-name>
<param-value system-property="tangosol.coherence.proxy.request.timeout"></param-value>
</init-param>
<init-param id="7">
<param-name>serializer</param-name>
<param-value></param-value>
</init-param>
<init-param id="8">
<param-name>guardian-timeout</param-name>
<param-value system-property="tangosol.coherence.proxy.guard.timeout"></param-value>
</init-param>
</init-params>
</service>
<service id="8">
<service-type>RemoteCache</service-type>
<service-component>RemoteCache</service-component>
<init-params>
<init-param id="1">
<param-name>initiator-config</param-name>
<param-value></param-value>
</init-param>
<init-param id="2">
<param-name>serializer</param-name>
<param-value></param-value>
</init-param>
</init-params>
</service>
<service id="9">
<service-type>RemoteInvocation</service-type>
<service-component>RemoteInvocation</service-component>
<init-params>
<init-param id="1">
<param-name>initiator-config</param-name>
<param-value></param-value>
</init-param>
<init-param id="2">
<param-name>serializer</param-name>
<param-value></param-value>
</init-param>
</init-params>
</service>
</services>
<filters>
<filter id="1">
<filter-name>gzip</filter-name>
<filter-class>com.tangosol.net.CompressionFilter</filter-class>
<init-params>
<init-param id="1">
<param-name>strategy</param-name>
<param-value>gzip</param-value>
</init-param>
<init-param id="2">
<param-name>level</param-name>
<param-value>default</param-value>
</init-param>
</init-params>
</filter>
<filter id="2">
<filter-name>symmetric-encryption</filter-name>
<filter-class>com.tangosol.net.security.PasswordBasedEncryptionFilter</filter-class>
<init-params>
<init-param id="1">
<param-name>password</param-name>
<param-value system-property="tangosol.coherence.security.password"></param-value>
</init-param>
</init-params>
</filter>
<filter id="3">
<filter-name>pkcs-encryption</filter-name>
<filter-class>com.tangosol.net.security.ClusterEncryptionFilter</filter-class>
<init-params>
<init-param id="1">
<param-name>store</param-name>
<param-value system-property="tangosol.coherence.security.keystore">./keystore.jks</param-value>
</init-param>
<init-param id="2">
<param-name>keyAlias</param-name>
<param-value>local</param-value>
</init-param>
<init-param id="3">
<param-name>keyPassword</param-name>
<param-value system-property="tangosol.coherence.security.password"></param-value>
</init-param>
</init-params>
</filter>
</filters>
</cluster-config>
<logging-config>
<destination system-property="tangosol.coherence.log">stdout</destination>
<logger-name system-property="tangosol.coherence.log.logger"></logger-name>
<severity-level system-property="tangosol.coherence.log.level">9</severity-level>
<message-format>{date}/{uptime} {product} {version} <{level}>
(thread={thread}, member={member}): {text}</message-format>
<character-limit system-property="tangosol.coherence.log.limit">1048576</character-limit>
</logging-config>
<configurable-cache-factory-config>
<class-name>com.tangosol.net.DefaultConfigurableCacheFactory</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value system-property="tangosol.coherence.cacheconfig">coherence-cache-config.xml</param-value>
</init-param>
</init-params>
</configurable-cache-factory-config>
<cache-factory-builder-config>
<class-name system-property="tangosol.coherence.cachefactorybuilder">com.tangosol.net.DefaultCacheFactoryBuilder</class-name>
</cache-factory-builder-config>
<management-config>
<!-- Note: Setting this value to 'local-only', 'remote-only or 'all' will
require the javax.management.* classes be available on the classpath. -->
<managed-nodes system-property="tangosol.coherence.management">all</managed-nodes>
<!-- Note: Setting this value to 'true' will require the Enterprise Edition
license. -->
<allow-remote-management system-property="tangosol.coherence.management.remote">true</allow-remote-management>
<refresh-policy system-property="tangosol.coherence.management.refresh.policy">refresh-ahead</refresh-policy>
<refresh-expiry system-property="tangosol.coherence.management.refresh.expiry">1s</refresh-expiry>
<refresh-timeout system-property="tangosol.coherence.management.refresh.timeout">250ms</refresh-timeout>
<read-only system-property="tangosol.coherence.management.readonly">false</read-only>
<default-domain-name>AGP Cache</default-domain-name>
<service-name>Management</service-name>
<server-factory>
<class-name system-property="tangosol.coherence.management.serverfactory"></class-name>
</server-factory>
<mbeans xml-override="{tangosol.coherence.mbeans /custom-mbeans.xml}">
<mbean id="1">
<mbean-query>java.lang:*</mbean-query>
<mbean-name>type=Platform</mbean-name>
<enabled system-property="tangosol.coherence.management.jvm.all">true</enabled>
</mbean>
</mbeans>
<reporter>
<configuration system-property="tangosol.coherence.management.report.configuration">reports/report-group.xml</configuration>
<autostart system-property="tangosol.coherence.management.report.autostart">false</autostart>
<distributed system-property="tangosol.coherence.management.report.distributed">false</distributed>
</reporter>
</management-config>
<security-config>
<enabled system-property="tangosol.coherence.security">false</enabled>
<login-module-name>Coherence</login-module-name>
<access-controller>
<class-name>com.tangosol.net.security.DefaultController</class-name>
<init-params>
<init-param id="1">
<param-type>java.io.File</param-type>
<param-value system-property="tangosol.coherence.security.keystore"></param-value>
</init-param>
<init-param id="2">
<param-type>java.io.File</param-type>
<param-value system-property="tangosol.coherence.security.permissions"></param-value>
</init-param>
</init-params>
</access-controller>
<callback-handler>
<class-name />
</callback-handler>
</security-config>
<license-config>
<edition-name system-property="tangosol.coherence.edition">GE</edition-name>
<license-mode system-property="tangosol.coherence.mode">dev</license-mode>
</license-config>
</coherence> -
I Do not have this problem with the WMP app. and it makes the view pain of iTunes look cluttered. along with the fact that the album that is in error
has no image associated with it, so that it looks even more unorganized.Steve MacGuire aka turingtest2 - iTunes & iPod Hints & Tips - Grouping Tracks Into Albums - http://www.samsoft.org.uk/iTunes/grouping.asp (older post on Apple Discussions http://discussions.apple.com/message.jspa?messageID=9910895)
Quick answer: Select all the tracks on the album, File > get info, and either give them all a single "album artist" or check the "compilation" flag. -
Hi,
I am trying to start coherence server on two different machines. I have modified tangosol-coherence-override-dev.xml for include ipaddress to WKA list (Followed the instruction from http://confluence.atlassian.com/display/CONFKB/How+to+Configure+a+Cluster+Without+Multicast+Traffic).
When I start the coherence server on two machines. Both machines coherences servers are not started, It waiting for another node to connect. Got the below error message
2010-12-26 14:16:11.469/280.351 Oracle Coherence GE n/a <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known node s to respond
Exception in thread "main" com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=ServiceMemberSet(
OldestMember=n/a
ActualMemberSet=MemberSet(Size=0, BitSetCount=0
MemberId/ServiceVersion/ServiceJoined/MemberState
Thanks in advance for your help.The request timeout exception occurred while starting a service.
The is a general document on metalink (document 845363.1) which talks about several potential reasons for service timeout.
However I suspect that there is a problem in the WKA configuration, as this error can be seen using an unreachable hostname in the <address> element -
"cluster.ClusteredHandlerFactory: Failed to initialize Coherence Cluster"
Hi,
I have a cluster with Admin, Proxy and MS1 on one server and MS2 ,MS3 on another server.
Below is the content of tangosol-coherence.xml file modified according to my env set up (The file is taken from Middleware1036/coherence_3.7/lib/coherence.jar and retained the content that is only required for the unicast configuration)
<cluster-config>
<member-identity>
<!--
Choose a unique name for the cluster, such that no other OSM instance in your
network would have the same name
-->
<cluster-name system-property="tangosol.coherence.cluster">ThirdCluster
</cluster-name>
</member-identity>
<unicast-listener>
<!--
Add more <socket-address>socket_address</socket-address> elements to well-known
addresses, one for each member in your OSM WebLogic cluster.
-->
<well-known-addresses>
<socket-address id="1">
<address>host1.example.com</address>
<port>31171</port>
</socket-address>
<socket-address id="2">
<address>host2.example.com</address>
<port>31172</port>
</socket-address>
<socket-address id="3">
<address>host2.example.com</address>
<port>31173</port>
</socket-address>
</well-known-addresses>
</unicast-listener>
</cluster-config>
After configuring the cluster domain with coherence, I made all the managed servers up with Admin and proxy and installed my application(Oracle communications order and service management product) into the cluster.
As per recommendation I need to restart all the servers to see my application osm.ear in active state.
Admin, proxy got restarted but while restarting the managed servers below is the error message I am encountering:
Hence please assist me on why is it saying "Failed to initialize the coherence cluster" .
####<Aug 6, 2012 6:20:04 AM PDT> <Error> <oms> <blr2230328> <ms1> <[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <oms-internal> <> <0000JZw4SMKB1FHpMs8Dye1G7wBy000001> <1344259204521> <BEA-000000> <cluster.ClusteredHandlerFactory: Failed to initialize Coherence cluster
com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.mslv.oms.handler.cluster.g.refresh(Unknown Source)
at oracle.communications.ordermanagement.listener.impl.a.a(Unknown Source)
at com.mslv.oms.handler.cluster.ClusteredHandlerFactory.<clinit>(Unknown Source)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at com.mslv.oms.security.HandlerFactory.b(Unknown Source)
at com.mslv.oms.security.HandlerFactory.startup(Unknown Source)
at com.mslv.oms.j2ee.LifecycleListener.postStart(Unknown Source)
at weblogic.application.internal.flow.BaseLifecycleFlow$PostStartAction.run(BaseLifecycleFlow.java:297)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.application.internal.flow.BaseLifecycleFlow$LifecycleListenerAction.invoke(BaseLifecycleFlow.java:199)
at weblogic.application.internal.flow.BaseLifecycleFlow.postStart(BaseLifecycleFlow.java:71)
at weblogic.application.internal.flow.TailLifecycleFlow.activate(TailLifecycleFlow.java:33)
at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:671)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:212)
at weblogic.application.internal.EarDeployment.activate(EarDeployment.java:59)
at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:161)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79)
at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicDeployment.java:184)
at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromServerLifecycle(BasicDeployment.java:361)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doActivate(DeploymentAdapter.java:51)
at weblogic.management.deploy.internal.DeploymentAdapter.activate(DeploymentAdapter.java:200)
at weblogic.management.deploy.internal.AppTransition$2.transitionApp(AppTransition.java:30)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
at weblogic.management.deploy.internal.ConfiguredDeployments.activate(ConfiguredDeployments.java:169)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:123)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:180)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:96)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)user768552 wrote:
Hi,
I have a cluster with Admin, Proxy and MS1 on one server and MS2 ,MS3 on another server.
Below is the content of tangosol-coherence.xml file modified according to my env set up (The file is taken from Middleware1036/coherence_3.7/lib/coherence.jar and retained the content that is only required for the unicast configuration)
<cluster-config>
<member-identity>
<!--
Choose a unique name for the cluster, such that no other OSM instance in your
network would have the same name
-->
<cluster-name system-property="tangosol.coherence.cluster">ThirdCluster
</cluster-name>
</member-identity>
<unicast-listener>
<!--
Add more <socket-address>socket_address</socket-address> elements to well-known
addresses, one for each member in your OSM WebLogic cluster.
-->
<well-known-addresses>
<socket-address id="1">
<address>host1.example.com</address>
<port>31171</port>
</socket-address>
<socket-address id="2">
<address>host2.example.com</address>
<port>31172</port>
</socket-address>
<socket-address id="3">
<address>host2.example.com</address>
<port>31173</port>
</socket-address>
</well-known-addresses>
</unicast-listener>
</cluster-config>
After configuring the cluster domain with coherence, I made all the managed servers up with Admin and proxy and installed my application(Oracle communications order and service management product) into the cluster.
As per recommendation I need to restart all the servers to see my application osm.ear in active state.
Admin, proxy got restarted but while restarting the managed servers below is the error message I am encountering:
Hence please assist me on why is it saying "Failed to initialize the coherence cluster" .
####<Aug 6, 2012 6:20:04 AM PDT> <Error> <oms> <blr2230328> <ms1> <[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <oms-internal> <> <0000JZw4SMKB1FHpMs8Dye1G7wBy000001> <1344259204521> <BEA-000000> <cluster.ClusteredHandlerFactory: Failed to initialize Coherence cluster
com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.mslv.oms.handler.cluster.g.refresh(Unknown Source)
at oracle.communications.ordermanagement.listener.impl.a.a(Unknown Source)
at com.mslv.oms.handler.cluster.ClusteredHandlerFactory.<clinit>(Unknown Source)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at com.mslv.oms.security.HandlerFactory.b(Unknown Source)
at com.mslv.oms.security.HandlerFactory.startup(Unknown Source)
at com.mslv.oms.j2ee.LifecycleListener.postStart(Unknown Source)
at weblogic.application.internal.flow.BaseLifecycleFlow$PostStartAction.run(BaseLifecycleFlow.java:297)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.application.internal.flow.BaseLifecycleFlow$LifecycleListenerAction.invoke(BaseLifecycleFlow.java:199)
at weblogic.application.internal.flow.BaseLifecycleFlow.postStart(BaseLifecycleFlow.java:71)
at weblogic.application.internal.flow.TailLifecycleFlow.activate(TailLifecycleFlow.java:33)
at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:671)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:212)
at weblogic.application.internal.EarDeployment.activate(EarDeployment.java:59)
at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:161)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79)
at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicDeployment.java:184)
at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromServerLifecycle(BasicDeployment.java:361)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doActivate(DeploymentAdapter.java:51)
at weblogic.management.deploy.internal.DeploymentAdapter.activate(DeploymentAdapter.java:200)
at weblogic.management.deploy.internal.AppTransition$2.transitionApp(AppTransition.java:30)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
at weblogic.management.deploy.internal.ConfiguredDeployments.activate(ConfiguredDeployments.java:169)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:123)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:180)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:96)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)Seems like, the OSM application is unable to join the cluster. Modify the WKA definition as below:
<well-known-addresses>
<socket-address id="1">
<address system-property="tangosol.coherence.wka1"></address>
<port system-property="tangosol.coherence.wka1.port"></port>
</socket-address>
</well-known-addresses>
Now, in the ServerStart of the ManagedServers running the OSM application add the following -Dtangosol.coherence.wka1=<> -Dtangosol.coherence.wka1.port=<>
Hope this helps!
Cheers,
NJ -
OrainstRoot.sh: Failure to promote local gpnp setup to other cluster nodes
I'm trying to build a 2 node cluster and everything appeared to be going swimmingly until the end of the 1st nodes running of the orainstRoot.sh script.
The following is the end of the output:
Disk Group OCR_VOTE created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 4e3f692529584f8bbf7f16146bd90346.
Successful addition of voting disk 728bed918cf54f6cbf904d37638c674b.
Successful addition of voting disk 8ac20793405d4fdcbfcafc7e311f877d.
Successfully replaced voting disk group with +OCR_VOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
1. ONLINE 4e3f692529584f8bbf7f16146bd90346 (ORCL:VOTE01) [OCR_VOTE]
2. ONLINE 728bed918cf54f6cbf904d37638c674b (ORCL:VOTE02) [OCR_VOTE]
3. ONLINE 8ac20793405d4fdcbfcafc7e311f877d (ORCL:VOTE03) [OCR_VOTE]
Located 3 voting disk(s).
Failed to rmtcopy "/tmp/fileLgKPGV" to "/u01/app/11.2.0/grid/gpnp/manifest.txt" for nodes {ilprevzedb01,ilprevzedb02}, rc=256
Failed to rmtcopy "/u01/app/11.2.0/grid/gpnp/ilprevzedb01/profiles/peer/profile.xml" to "/u01/app/11.2.0/grid/gpnp/profiles/peer/profile.xml" for nodes {ilprevzedb01,ilprevzedb02}, rc=256
rmtcopy aborted
Failed to promote local gpnp setup to other cluster nodes at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6504.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
Has anyone run into this problem and found a solution?
Thanks in advance!Ok, for everyone out there, I resolved the issue. Hopefully this will help others encountering the same problem.
It turns out that when the OS was installed, iptables firewall was enabled. This will cause havoc with the installer scripts.
My first inkling should have been when the installer stalled at 65% trying to copy home directories between nodes, the first time I ran through the installer.
At that time, Googling around found that iptables might be the problem and indeed it was running, so I just did a 'service iptables stop' WITHOUT REBOOTING THE NODES and re-ran the installer.
Well, it looks as though NOT REBOOTING THE NODES doesn't quite cut it. I then did a 'chkconfig iptables off' and REBOOTED BOTH NODES.
Oracle support simply provided me with: How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation (Doc ID 942166.1), which didn't really work all that well, lots of failures, errors, etc. So I just deleted the 11.2.0 directory and tried running the installer again.
This time the install went through without problems.
Thanks! -
Having issue with start weblogic cluster with tangosol cluster
Hi,
Oracle Coherence Version 3.3.1/389p1
Grid Edition: Development mode
We are using Weblogic 8.1.5 with Tangosol 3.3.1 on Linux servers.
And we added the initializing logic in the servlet's init() method to get all NamedCaches and put into the ServletContext.
When we start weblogic cluster, the first weblogic member will startup successfully with following messages :
<Nov 7, 2007 10:12:30 AM EST> <Info> <HTTP> <BEA-101047> <[2007-11-07 10ServletContext(id=259640596,name=clusterqa,context-path=)] initObjects: init>:12:31.565 Oracle Coherence 3.3.1/389p1 <Info> (thread=Main Thread, member=n/a): Loaded operational configuration from resource "zip:/home/server/clusterqa/wls81/DOCVIEW/docqa1/.wlnotdelete/extract/docqa1_DOC_clusterqa/jarfiles/WEB-INF/lib/coherence.jar!/tangosol-coherence.xml"
2007-11-07 10:12:31.598 Oracle Coherence 3.3.1/389p1 <Info> (thread=Main Thread, member=n/a): Loaded operational overrides from file "/home/www/WEB-INF/lib/tangosol-coherence-override.xml"
Oracle Coherence Version 3.3.1/389p1
Grid Edition: Development mode
Copyright (c) 2000-2007 Oracle. All rights reserved.
2007-11-07 10:12:31.938 Oracle Coherence GE 3.3.1/389p1 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from file "/home/www/WEB-INF/lib/pub-search-cache-config.xml"
2007-11-07 10:12:31.983 Oracle Coherence GE 3.3.1/389p1 <Info> (thread=Main Thread, member=n/a): sun.misc.AtomicLong is not supported on this JVM; using a synchronized counter. Though safe to ignore, you may upgrade to BEA's 1.5 JVM to fix this issue.
2007-11-07 10:12:33.267 Oracle Coherence GE 3.3.1/389p1 <Warning> (thread=Main Thread, member=n/a): UnicastUdpSocket failed to set receive buffer size to 1428 packets (2096304 bytes); actual size is 89 packets (131071 bytes). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.
2007-11-07 10:12:34.118 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2007-11-07 10:12:37.508 Oracle Coherence GE 3.3.1/389p1 <Info> (thread=Cluster, member=n/a): Created a new cluster with Member(Id=1, Timestamp=2007-11-07 10:12:33.323, Address=10.5.176.86:8088, MachineId=48982, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=2) UID=0x0A05B056000001161AAB782BBF561F98
2007-11-07 10:12:37.736 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
2007-11-07 10:12:38.168 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 1
<Nov 7, 2007 10:12:38 AM EST> <Info> <HTTP> <BEA-101047> <[ServletContext(id=259640596,name=clusterqa,context-path=)] xslProcessor: init>
But trying to start the second weblogic member server, the startup process is stucked after tangosol cache initialization and the second weblogic member server never up running. Please see following messages :
<Nov 7, 2007 9:49:38 AM EST> <Info> <HTTP> <BEA-101047> <[ServletContext(id=153019550,name=clusterqa,context-path=)] initDSNames: init>
<Nov 7, 2007 9:49:42 AM EST> <Info> <HTTP> <BEA-101047> <[ServletContext(id=153019550,name=clusterqa,context-path=)] initObjects: init>
2007-11-07 09:49:43.156 Oracle Coherence 3.3.1/389p1 <Info> (thread=Main Thread, member=n/a): Loaded operational configuration from resource "zip:/home/server/clusterqa/wls81/DOCVIEW/docqa2/.wlnotdelete/extract/docqa2_DOC_clusterqa/jarfiles/WEB-INF/lib/coherence.jar!/tangosol-coherence.xml"
2007-11-07 09:49:43.188 Oracle Coherence 3.3.1/389p1 <Info> (thread=Main Thread, member=n/a): Loaded operational overrides from file "/home/www/WEB-INF/lib/tangosol-coherence-override.xml"
Oracle Coherence Version 3.3.1/389p1
Grid Edition: Development mode
Copyright (c) 2000-2007 Oracle. All rights reserved.
2007-11-07 09:49:43.528 Oracle Coherence GE 3.3.1/389p1 <Info> (thread=Main Thread, member=n/a): Loaded cache configuration from file "/home/www/WEB-INF/lib/pub-search-cache-config.xml"
2007-11-07 09:49:43.571 Oracle Coherence GE 3.3.1/389p1 <Info> (thread=Main Thread, member=n/a): sun.misc.AtomicLong is not supported on this JVM; using a synchronized counter. Though safe to ignore, you may upgrade to BEA's 1.5 JVM to fix this issue.
2007-11-07 09:49:44.829 Oracle Coherence GE 3.3.1/389p1 <Warning> (thread=Main Thread, member=n/a): UnicastUdpSocket failed to set receive buffer size to 1428 packets (2096304 bytes); actual size is 89 packets (131071 bytes). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.
2007-11-07 09:49:45.419 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2007-11-07 09:49:45.555 Oracle Coherence GE 3.3.1/389p1 <Info> (thread=Cluster, member=n/a): Failed to satisfy the variance: allowed=16, actual=47
2007-11-07 09:49:45.555 Oracle Coherence GE 3.3.1/389p1 <Info> (thread=Cluster, member=n/a): Increasing allowable variance to 19
2007-11-07 09:49:46.040 Oracle Coherence GE 3.3.1/389p1 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Timestamp=2007-11-07 09:49:45.69, Address=10.5.176.85:8088, MachineId=48981, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=2) joined cluster with senior Member(Id=1, Timestamp=2007-11-07 09:45:10.205, Address=10.5.176.86:8088, MachineId=48982, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=2)
Could you please explain why it happens, and what should I do to resolve this issues ?
Many Thanks,
BingHi, Gene
Thank you for the response. I will send you our full log files and thread dumps.
I just want to give you more details about our cases :
1. This only happened without starting cache servers (com.tangosol.net.DefaultCacheServer).
2. And our application which are running on weblogic cluster will just call "CacheFactory.getCache("XXX")", and running as the Tangosol DataClient.
3. All weblogic member servers will be up running successfully if our cache servers are up running.
Also I tried to test another case :
Suppose all weblogic instances and cache server instances are up running. Now I trying to restart (kill weblogic instance process and restart) one of the weblogic member, It will up running successfully only if add some sleep times after killing weblogic processes and restarting it. Looks like tangosol cluster need certain time to aware the member has left cluster, then the restart process will be successful.
Questions :
1. Should we start our weblogic cluster only after cache server cluster is up running ?
2. How do we decide how many time we should wait before start new process to join the cache cluster ?
Could you please help to explain this one for me and let us if there anyway we can do to avoid the problem.
Many Thanks !!!
Bing -
Can I create different Coherence nodes in the same cluster with defferent?
Can I create different Coherence nodes in the same cluster with defferent cache-config.xml file ?
Can a cache be distributed in these deffirent nodes?Yes. You can create different Coherence nodes in the same cluster with defferent cache-config.xml files as long as you use the same tangosol-coherence.xml file and the same tangosol-coherence-override.xml file. But you cannot store the cache data in the different nodes (started with different cache-config file). In other word, a node only create caches in their own's modes which are started with the same cache-config.xml file.
See the following demo:
I start a cache server using the cache config file examples-cache-server.xml. Then I start a storage-disabled cache console (cache client) using the cache config file coherence-cache-config.xml. Both of them using the same tangosol-coherence.xml file and the same tangosol-coherence-override.xml file.
The cache server uses a cache service PartitionedPofCache. But the client side is using the Distributedcache service. The cluster address is same 224.3.5.2.
The cluster name is also samme. They know each other.
D:\coherence\lib>D:\examples\java\bin\run-cache-server.cmd
D:\coherence\lib>D:\examples\java\bin\run-cache-server.cmd
The system cannot find the file D:\coherence.
The system cannot find the file C:\Oracle\Middleware\jdk160_11.
2009-12-22 12:09:31.400/4.987 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational configurat
ion from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2009-12-22 12:09:31.450/5.037 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational overrides
from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2009-12-22 12:09:31.470/5.057 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/tangosol-coherence-override.xml" is not specified
2009-12-22 12:09:31.540/5.127 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.5.2/463
Grid Edition: Development mode
Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
2009-12-22 12:09:33.864/7.451 Oracle Coherence GE 3.5.2/463 <Info> (thread=main, member=n/a): Loaded cache configuration
from "file:/D:/examples/java/resource/config/examples-cache-config.xml"
2009-12-22 12:09:39.983/13.570 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Service Cluster joined t
he cluster with senior service member n/a
2009-12-22 12:09:43.187/16.774 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): Created a new cluster
"cluster:0xD3FB" with Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Locatio
n=process:144, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1) UID=0xC0A8085000
000125B75D888C60501F98
2009-12-22 12:09:43.508/17.095 Oracle Coherence GE 3.5.2/463 <D5> (thread=Invocation:Management, member=1): Service Mana
gement joined the cluster with senior service member 1
2009-12-22 12:09:46.582/20.169 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache:PartitionedPofCache, member=1
): Service PartitionedPofCache joined the cluster with senior service member 1
2009-12-22 12:09:46.672/20.259 Oracle Coherence GE 3.5.2/463 <Info> (thread=DistributedCache:PartitionedPofCache, member
=1): Loading POF configuration from resource "file:/D:/examples/java/resource/config/examples-pof-config.xml"
2009-12-22 12:09:46.702/20.289 Oracle Coherence GE 3.5.2/463 <Info> (thread=DistributedCache:PartitionedPofCache, member
=1): Loading POF configuration from resource "jar:file:/D:/coherence/lib/coherence.jar!/coherence-pof-config.xml"
2009-12-22 12:09:47.734/21.321 Oracle Coherence GE 3.5.2/463 <Info> (thread=main, member=1): Started DefaultCacheServer.
SafeCluster: Name=cluster:0xD3FB
Group{Address=224.3.5.2, Port=35463, TTL=4}
MasterMemberSet
ThisMember=Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process
:144, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=proce
ss:144, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process:144, Rol
e=CoherenceServer)
RecycleMillis=120000
RecycleSet=MemberSet(Size=0, BitSetCount=0
Services
TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=192.168.8.80:8088}, Connections=[]}
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
DistributedCache{Name=PartitionedPofCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCo
unt=1, AssignedPartitions=257, BackupPartitions=0}
2009-12-22 12:12:29.737/183.324 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=20
09-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=process:1188, Role=CoherenceConsole) joined
Cluster with senior member 1
2009-12-22 12:12:30.498/184.085 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 2 joined Service M
anagement with senior member 1
2009-12-22 12:12:31.860/185.447 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): TcpRing: connecting to me
mber 2 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/192.168.8.80,port=8089,localport=2463]}
2009-12-22 12:12:51.338/204.925 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 2 joined Service D
istributedCache with senior member 2The following command starts a cache client.
D:\coherence\bin>coherence.cmd
D:\coherence\bin>coherence.cmd
** Starting storage disabled console **
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Server VM (build 11.0-b16, mixed mode)
2009-12-22 12:12:21.054/3.425 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational configurat
ion from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2009-12-22 12:12:21.355/3.726 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational overrides
from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2009-12-22 12:12:21.365/3.736 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/tangosol-coherence-override.xml" is not specified
2009-12-22 12:12:21.415/3.786 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.5.2/463
Grid Edition: Development mode
Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
2009-12-22 12:12:29.316/11.687 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Service Cluster joined t
he cluster with senior service member n/a
2009-12-22 12:12:29.356/11.727 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): Failed to satisfy the
variance: allowed=16, actual=20
2009-12-22 12:12:29.356/11.727 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): Increasing allowable v
ariance to 17
2009-12-22 12:12:29.807/12.178 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Time
stamp=2009-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=process:1188, Role=CoherenceConsole,
Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1) joined cluster "cluster:0xD3FB" with senior Member(I
d=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process:144, Role=CoherenceS
erver, Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1)
2009-12-22 12:12:29.977/12.348 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Member 1 joined Service
Management with senior member 1
2009-12-22 12:12:29.977/12.348 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Member 1 joined Service
PartitionedPofCache with senior member 1
2009-12-22 12:12:30.578/12.949 Oracle Coherence GE 3.5.2/463 <D5> (thread=Invocation:Management, member=2): Service Mana
gement joined the cluster with senior service member 1
SafeCluster: Name=cluster:0xD3FB
Group{Address=224.3.5.2, Port=35463, TTL=4}
MasterMemberSet
ThisMember=Member(Id=2, Timestamp=2009-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=proces
s:1188, Role=CoherenceConsole)
OldestMember=Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=proce
ss:144, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=2, BitSetCount=2
Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process:144, Rol
e=CoherenceServer)
Member(Id=2, Timestamp=2009-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=process:1188, R
ole=CoherenceConsole)
RecycleMillis=120000
RecycleSet=MemberSet(Size=0, BitSetCount=0
Services
TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=192.168.8.80:8089}, Connections=[]}
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
Map (?):
2009-12-22 12:12:49.505/31.906 Oracle Coherence GE 3.5.2/463 <Info> (thread=main, member=2): Loaded cache configuration
from "jar:file:/D:/coherence/lib/coherence.jar!/coherence-cache-config.xml"
2009-12-22 12:12:51.358/33.729 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=2): Service Distribut
edCache joined the cluster with senior service member 2
<distributed-scheme>
<!--
To use POF serialization for this partitioned service,
uncomment the following section
<serializer>
<class-
name>com.tangosol.io.pof.ConfigurablePofContext</class-
name>
</serializer>
-->
<scheme-name>example-distributed</scheme-name>
<service-name>DistributedCache</service-name>
<backing-map-scheme>
<local-scheme>
<scheme-ref>example-binary-backing-map</scheme-ref>
</local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>But when I try to store data into cache from the client side, it report error message: it's staorage-disabled. It shows that this cache console cannot store the data in the existing cache server because then using different cache config files.
Map (ca3): cache ca2
<distributed-scheme>
<!--
To use POF serialization for this partitioned service,
uncomment the following section
<serializer>
<class-
name>com.tangosol.io.pof.ConfigurablePofContext</class-
name>
</serializer>
-->
<scheme-name>example-distributed</scheme-name>
<service-name>DistributedCache</service-name>
<backing-map-scheme>
<local-scheme>
<scheme-ref>example-binary-backing-map</scheme-ref>
</local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
Map (ca2): put 1 one
2009-12-22 14:00:04.999/6467.370 Oracle Coherence GE 3.5.2/463 <Error> (thread=main, member=2):
java.lang.RuntimeException: Storage is not configured
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.onMissing
Storage(DistributedCache.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureReq
uestTarget(DistributedCache.CDB:34)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.put(Distr
ibutedCache.CDB:22)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.put(Distr
ibutedCache.CDB:1)
at com.tangosol.util.ConverterCollections$ConverterMap.put(ConverterCollections.java:1541)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.put(Distrib
utedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.put(SafeNamedCache.CDB:1)
at com.tangosol.coherence.component.application.console.Coherence.processCommand(Coherence.CDB:581)
at com.tangosol.coherence.component.application.console.Coherence.run(Coherence.CDB:39)
at com.tangosol.coherence.component.application.console.Coherence.main(Coherence.CDB:3)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.tangosol.net.CacheFactory.main(CacheFactory.java:1400) -
Cluster Setup Support Rules fails with Warnings
Hi,
I'm in the process of installing our ECC production server in a clustered environment on windows server 2008 with SQL server 2008 as our database.
The MSCS cluster at the OS level has been configured successfully. When I start the SQL server installation as mentioned in the installation guide, the Setup Support Rules fails with following warnings.
1. The setup support file prompts me to set the MSDTC service as a clustered one but it was not mentioned anywhere in the installation guide that the MSDTC service needs to be clustered. Do I need to cluster that service with seperate network ip address and cluster group ?
2. The Network binding order issues a warning even though the adapters are set up in the right order as mentioned in the installation guide.
3. Eventhough there is a internet connectivity available in the server, the "Microsoft .Net Application Security" check issues a warning.
NB: As per the cluster validation report, the clustering at the OS level seems to be fine.
Please provide me your valuable suggestions to rectify these warnings. Can I ignore these warnings and proceed with the installation of SQL Server ?
Regards,
Varadharajan MHi,
Microsoft Distributed Transaction Coordinator
Is is user by older versions of Mssql database.
From a SAP perspective you don't need this service.
You can ignore setup request for this service.
As per the reply, I understand that the MSDTC service is not mandatory to cluster in Windows Server 2008/MS SQL Server 2008. But when i try to install the SQL DB, the Set support rule ends up with warnings saying that the MSDTC service should be clustered. Can I ignore the warnings and proceed with the installation. Can anyone confirm this please ?
Also please confirm whether we need to have the db cluster group in the DNS before initiating the SQL Server Set up.
Regards,
Varadharajan M -
Issue listeneing queue from Weblogic Cluster server with multiple managed server
Haveing issue listeneing queue from Weblogic Cluster server with multiple managed server.
Weblogic Cluster structure is like
Weblogic Cluster01
--ManagedServer01(http://server01.myhost.com:7001)
--ManagedServer02(http://server02.myhost.com:7001)
JMS Servers
JMSserver01 targeting: ManagedServer01
JMSserver02 targeting: ManagedServer02
JMSmodule
ConnectionFactory01 targeting:JMSserver01,JMSserver02
UDQueue01 targeting:JMSserver01,JMSserver02
Uniform Distributed Queue in Monitoring tab showing like this
mysystemmodule!JMSserver01@UDQueue01
mysystemmodule!JMSserver02@UDQueue01
So when I am sending message to any Host(by specifying the provider URL) its distributing equally on both server like
mysystemmodule!JMSserver01@UDQueue01 10
mysystemmodule!JMSserver02@UDQueue01 10
But when try to listen message from these queue, it is listening from one server, for which URL given to connect.
mysystemmodule!JMSserver01@UDQueue01 0
mysystemmodule!JMSserver02@UDQueue01 10
untill I connect to other server by giveing its URL, will not able to access other message left on the queue.
Solutions that tried
1) we have tried give both server URL coma sparated in provider URL
we need to configur same scenario for 5 managed server with 3 listener on other servers.
Do any one have solution for this.You need to have:
1. Consumers connected to each UDQ member
OR
2. If no consumers in some of the members is expected, you can configure Forward Delay (specify the amount of time, in seconds, that a queue member with messages, but with no consumers, will wait before forwarding its messages to other queue members that do have consumers):
http://docs.oracle.com/cd/E12839_01/apirefs.1111/e13952/taskhelp/jms_modules/distributed_queues/ConfigureUDQGeneral.html
For example you can set it to 10 (10s)
Additional Information here:
http://docs.oracle.com/cd/E23943_01/web.1111/e13727/dds.htm#i1314228
http://docs.oracle.com/cd/E23943_01/apirefs.1111/e13951/mbeans/DistributedQueueBean.html?skipReload=true#ForwardDelay
How Does JMS Load Balancing Work with Distributed Queues and Uniform Distributed Queues? (Doc ID 827294.1)
I hope this helps
Best Regards
Luz -
Urgent! Node keep disconnecting from Coherence Cluster
The system consists of 4 standalone cache servers with local storage set to true and 14 other embedded nodes started with different web apps on tomcat with local storage set to false.
When the servers are started after a new deployment, sometimes it would just work, but most times some random tomcat server will stuck in the following pattern.
First it would successful start the cluster service and join an existing cluster.
Oracle Coherence Version 3.5.1/461
Grid Edition: Development mode
Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
2012-07-18 12:24:33.335/31.845 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2012-07-18 12:24:33.550/32.060 Oracle Coherence GE 3.5.1/461 <Info> (thread=Cluster, member=n/a): This Member(Id=8, Timestamp=2012-07-18 12:24:33.347, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:20831, Role=ApacheCatalinaStartupBootstrap, Edition=Grid Edition, Mode=Development, CpuCount=24, SocketCount=24) joined cluster "DEV1" with senior Member(Id=10, Timestamp=2012-07-18 09:39:44.861, Address=10.34.32.101:8090, MachineId=2149, Location=machine:dev1ssapp3,process:27796, Role=ApacheCatalinaStartupBootstrap, Edition=Grid Edition, Mode=Development, CpuCount=64, SocketCount=64)
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=1, Timestamp=2012-07-18 12:22:14.231, Address=10.34.32.107:8090, MachineId=2155, Location=machine:dev1sxapp2,process:1278, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=2, Timestamp=2012-07-18 12:22:14.331, Address=10.34.32.106:8089, MachineId=2154, Location=machine:dev1sxapp1,process:6549, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=3, Timestamp=2012-07-18 12:22:55.086, Address=10.34.32.106:8088, MachineId=2154, Location=machine:dev1sxapp1,process:23083, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=4, Timestamp=2012-07-18 12:22:56.799, Address=10.34.32.107:8088, MachineId=2155, Location=machine:dev1sxapp2,process:19624, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=5, Timestamp=2012-07-18 12:24:31.869, Address=10.34.32.106:8090, MachineId=2154, Location=machine:dev1sxapp1,process:24411, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=6, Timestamp=2012-07-18 12:24:33.084, Address=10.34.32.101:8088, MachineId=2149, Location=machine:dev1ssapp3,process:28932, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.555/32.065 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=14, Timestamp=2012-07-18 09:40:50.645, Address=10.34.32.104:8090, MachineId=2152, Location=machine:dev1ssapp4,process:17697, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.556/32.066 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=17, Timestamp=2012-07-18 10:35:16.722, Address=10.34.32.104:8093, MachineId=2152, Location=machine:dev1ssapp4,process:19365, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.556/32.066 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=18, Timestamp=2012-07-18 10:38:47.714, Address=10.34.32.101:8093, MachineId=2149, Location=machine:dev1ssapp3,process:29887, Role=ApacheCatalinaStartupBootstrap) joined Cluster with senior member 10
2012-07-18 12:24:33.563/32.073 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 10 joined Service Management with senior member 10
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 10
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 2 joined Service Management with senior member 10
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 4 joined Service Management with senior member 10
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 17 joined Service Management with senior member 10
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 17 joined Service PFExpiryDistributedCache with senior member 17
2012-07-18 12:24:33.566/32.076 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 17 joined Service SsoRuleEntryDistributedCache with senior member 17
2012-07-18 12:24:33.567/32.077 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 3 joined Service Management with senior member 10
2012-07-18 12:24:33.567/32.077 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 18 joined Service Management with senior member 10
2012-07-18 12:24:33.567/32.077 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 18 joined Service PFExpiryDistributedCache with senior member 17
2012-07-18 12:24:33.567/32.077 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 18 joined Service SsoRuleEntryDistributedCache with senior member 17
2012-07-18 12:24:33.568/32.078 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 14 joined Service Management with senior member 10
2012-07-18 12:24:33.568/32.078 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 5 joined Service Management with senior member 10
2012-07-18 12:24:33.579/32.089 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 6 joined Service Management with senior member 10
Then it started getting heartbeat overdue message and cluster stopped:
2012-07-18 12:37:20.717/799.227 Oracle Coherence GE 3.5.1/461 <Info> (thread=PacketListenerN, member=8): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2012-07-18 12:37:29.916/808.426 Oracle Coherence GE 3.5.1/461 <Info> (thread=PacketListenerN, member=8): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2012-07-18 12:37:59.291/837.801 Oracle Coherence GE 3.5.1/461 <Error> (thread=PacketListenerN, member=8): Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: Unable to refresh sockets: [UnicastUdpSocket{State=STATE_OPEN, address:port=10.34.32.107:8089}, MulticastUdpSocket{State=STATE_OPEN, address:port=237.0.0.1:40109, InterfaceAddress=10.34.32.107, TimeToLive=4}, TcpSocketAccepter{State=STATE_OPEN, ServerSocket=10.34.32.107:8089}]; last failed socket: MulticastUdpSocket{State=STATE_OPEN, address:port=237.0.0.1:40109, InterfaceAddress=10.34.32.107, TimeToLive=4}
at com.tangosol.coherence.component.net.Cluster$SocketManager.refreshSockets(Cluster.CDB:91)
at com.tangosol.coherence.component.net.Cluster$SocketManager$MulticastUdpSocket.onInterruptedIOException(Cluster.CDB:9)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:33)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: Receive timed out
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
at java.net.DatagramSocket.receive(DatagramSocket.java:725)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
2012-07-18 12:37:59.291/837.801 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=8): Service Cluster left the cluster
2012-07-18 12:37:59.293/837.803 Oracle Coherence GE 3.5.1/461 <D5> (thread=Invocation:Management, member=8): Service Management left the cluster
2012-07-18 12:37:59.293/837.803 Oracle Coherence GE 3.5.1/461 <D5> (thread=ReplicatedCache:HibernateReplicatedCache, member=8): Service HibernateReplicatedCache left the cluster
Then it started getting messages from various nodes about the existing cluster:
2012-07-18 12:40:02.862/961.372 Oracle Coherence GE 3.5.1/461 <Info> (thread=queue://authenticationService.logonEvent.consumer-2, member=n/a): Restarting cluster
2012-07-18 12:40:02.891/961.401 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2012-07-18 12:40:20.167/978.677 Oracle Coherence GE 3.5.1/461 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2012-07-18 12:40:02.867, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:20831, Role=ApacheCatalinaStartupBootstrap) has been attempting to join the cluster at address 237.0.0.1:40109 with TTL 4 for 17 seconds without success; this could indicate a mis-configured TTL value, or it may simply be the result of a busy cluster or active failover.
2012-07-18 12:40:20.168/978.678 Oracle Coherence GE 3.5.1/461 <Warning> (thread=Cluster, member=n/a): Received a discovery message that indicates the presence of an existing cluster:
Message "NewMemberAnnounceWait"
FromMember=Member(Id=4, Timestamp=2012-07-18 12:22:56.799, Address=10.34.32.107:8088, MachineId=2155, Location=machine:dev1sxapp2,process:19624, Role=ApacheCatalinaStartupBootstrap)
FromMessageId=0
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=9
ToPollId=0
Poll=null
Packets
[000]=Broadcast{PacketType=0x0DDF00D2, ToId=0, FromId=4, Direction=Incoming, ReceivedMillis=12:40:20.167, MessageType=9, MessagePartCount=1, MessagePartIndex=0, Body=0x00000001389AE63E1F0A22206B00000000000000000000000040001
F980000086B000405011818044445563140400A64657631737861707032053139363234401E417061636865436174616C696E6153746172747570426F6F7473747261700001000001389AF5E6330A22206B00000000000000000000000040001F990000086B000005011818044445563140
400A64657631737861707032053230383331401E417061636865436174616C696E6153746172747570426F6F74737472617000000001389AE6376E0A22206A00000000000000000000000040001F980000086A000305011818044445563140400A646576317378617070310532333038334
01E417061636865436174616C696E6153746172747570426F6F74737472617000, Body.length=287}
Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_ANNOUNCE), Id=0, Version=3.5}
ToMemberSet=null
NotifySent=false
ToMember=Member(Id=0, Timestamp=2012-07-18 12:40:02.867, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:20831, Role=ApacheCatalinaStartupBootstrap)
SeniorMember=Member(Id=3, Timestamp=2012-07-18 12:22:55.086, Address=10.34.32.106:8088, MachineId=2154, Location=machine:dev1sxapp1,process:23083, Role=ApacheCatalinaStartupBootstrap)
Then it failed to connect to the cluster:
2012-07-18 12:40:33.187/991.697 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster left the cluster
2012-07-18 12:40:33.190/991.700 Oracle Coherence GE 3.5.1/461 <Error> (thread=queue://authenticationService.logonEvent.consumer-2, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException:
Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=ServiceMemberSet(
OldestMember=n/a
ActualMemberSet=MemberSet(Size=0, BitSetCount=0
MemberId/ServiceVersion/ServiceJoined/ServiceLeaving
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:38)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:366)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:7)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:27)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:1011)
at com.tangosol.coherence.hibernate.CoherenceCacheProvider.nextTimestamp(CoherenceCacheProvider.java:58)
at org.hibernate.cache.impl.bridge.RegionFactoryCacheProviderBridge.nextTimestamp(RegionFactoryCacheProviderBridge.java:93)
at org.hibernate.impl.SessionFactoryImpl.openSession(SessionFactoryImpl.java:652)
at org.hibernate.ejb.EntityManagerImpl.getRawSession(EntityManagerImpl.java:111)
at org.hibernate.ejb.EntityManagerImpl.getSession(EntityManagerImpl.java:91)
at org.hibernate.ejb.AbstractEntityManagerImpl.setDefaultProperties(AbstractEntityManagerImpl.java:250)
at org.hibernate.ejb.AbstractEntityManagerImpl.postInit(AbstractEntityManagerImpl.java:162)
at org.hibernate.ejb.EntityManagerImpl.<init>(EntityManagerImpl.java:84)
at org.hibernate.ejb.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:112)
at org.hibernate.ejb.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:107)
at org.springframework.orm.jpa.JpaTransactionManager.createEntityManagerForTransaction(JpaTransactionManager.java:399)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:321)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:335)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
at java.lang.Thread.run(Thread.java:662)
2012-07-18 12:40:33.216/991.726 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2012-07-18 12:40:50.398/1008.908 Oracle Coherence GE 3.5.1/461 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2012-07-18 12:40:33.194, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:20831, Role=ApacheCatalinaStartupBootstrap) has been attempting to join the cluster at address 237.0.0.1:40109 with TTL 4 for 17 seconds without success; this could indicate a mis-configured TTL value, or it may simply be the result of a busy cluster or active failover.
This particular jvm then start go into this kind of loop, receive a lot of messages from other nodes about the exist cluster but failed to join.
I have ran the MulticastTest and Datagram test which didn't reveal any obvious network issue. What should I do next?
JVM is 1.6.0_31
Thanks a lot in advance, any help will be greatly appreciated.I correlated the log with all servers and found the issue might be due to some member it is connected with actually was being restarted.
Server 1:
- starts as member 23 and discovered the existing cluster and joined it. Then a lot of messages on server1 with all different members joining the cluster with different member id.
- Then it found some member failed to respond:
2012-07-30 22:00:25.371/34.325 Oracle Coherence GE 3.5.1/461 <D6> (thread=PacketPublisher, member=n/a): Member(Id=5, Timestamp=2012-07-30 15:35:09.735, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:21324, Role=ApacheCatalinaStartupBootstrap) has failed to respond to 17 packets; declaring this member as paused.
- Then it's requesting the departure confirmation for member 5:
2012-07-30 22:00:52.042/60.996 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=n/a): Timeout while delivering a packet Directed{PacketType=0x0DDF00D5, ToId=0, FromId=23, Direction=Outgoing, SentCount=145, SentMillis=22:00:51.832, ToMemberSet=[5(1)], ServiceId=0, MessageType=16, FromMessageId=6, ToMessageId=0, MessagePartCount=1, MessagePartIndex=0, NackInProgress=false, ResendScheduled=22:00:52.32, Timeout=22:00:51.849, PendingResendSkips=0, DeliveryState=unsent, Body=0x0000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000..., Body.length=1398}; requesting the departure confirmation for Member(Id=5, Timestamp=2012-07-30 15:35:09.735, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:21324, Role=ApacheCatalinaStartupBootstrap)
by MemberSet(Size=2, BitSetCount=2
Member(Id=1, Timestamp=2012-07-27 10:46:51.616, Address=10.34.32.101:8088, MachineId=2149, Location=machine:dev1ssapp3,process:1, Role=CoherenceServer)
Member(Id=12, Timestamp=2012-07-30 22:00:43.132, Address=10.34.32.101:8091, MachineId=2149, Location=machine:dev1ssapp3,process:7803, Role=ApacheCatalinaStartupBootstrap)
- Then the member set confirmed the departure however at the same time, service cluster also left.
2012-07-30 22:00:52.046/61.000 Oracle Coherence GE 3.5.1/461 <Info> (thread=Cluster, member=n/a): Member departure confirmed by MemberSet(Size=1, BitSetCount=2
Member(Id=12, Timestamp=2012-07-30 22:00:43.132, Address=10.34.32.101:8091, MachineId=2149, Location=machine:dev1ssapp3,process:7803, Role=ApacheCatalinaStartupBootstrap)
); removing Member(Id=5, Timestamp=2012-07-30 15:35:09.735, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:21324, Role=ApacheCatalinaStartupBootstrap)
2012-07-30 22:00:52.046/61.000 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=5, Timestamp=2012-07-30 22:00:52.046, Address=10.34.32.107:8089, MachineId=2155, Location=machine:dev1sxapp2,process:21324, Role=ApacheCatalinaStartupBootstrap) left Cluster with senior member 1
2012-07-30 22:00:52.049/61.003 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster left the cluster
- Then the timeout during service start hence application fails to start
2012-07-30 22:00:52.051/61.005 Oracle Coherence GE 3.5.1/461 <Error> (thread=main, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=ServiceMemberSet(
OldestMember=n/a
ActualMemberSet=MemberSet(Size=15, BitSetCount=2
Member(Id=1, Timestamp=2012-07-27 10:46:51.616, Address=10.34.32.101:8088, MachineId=2149, Location=machine:dev1ssapp3,process:1, Role=CoherenceServer)
Member(Id=2, Timestamp=2012-07-27 10:47:12.122, Address=10.34.32.101:8089, MachineId=2149, Location=machine:dev1ssapp3,process:2, Role=CoherenceServer)
Member(Id=3, Timestamp=2012-07-27 10:48:02.603, Address=10.34.32.104:8088, MachineId=2152, Location=machine:dev1ssapp4,process:1, Role=CoherenceServer)
Member(Id=4, Timestamp=2012-07-27 10:48:04.76, Address=10.34.32.104:8089, MachineId=2152, Location=machine:dev1ssapp4,process:2, Role=CoherenceServer)
Member(Id=8, Timestamp=2012-07-30 14:27:07.382, Address=10.34.32.101:8090, MachineId=2149, Location=machine:dev1ssapp3,process:23727, Role=ApacheCatalinaStartupBootstrap)
Member(Id=9, Timestamp=2012-07-30 22:00:28.596, Address=10.34.32.101:8092, MachineId=2149, Location=machine:dev1ssapp3,process:7619, Role=ApacheCatalinaStartupBootstrap)
Member(Id=10, Timestamp=2012-07-30 14:34:27.573, Address=10.34.32.104:8090, MachineId=2152, Location=machine:dev1ssapp4,process:25219, Role=ApacheCatalinaStartupBootstrap)
Member(Id=11, Timestamp=2012-07-30 22:00:41.609, Address=10.34.32.107:8088, MachineId=2155, Location=machine:dev1sxapp2,process:17632, Role=ApacheCatalinaStartupBootstrap)
Member(Id=12, Timestamp=2012-07-30 22:00:43.132, Address=10.34.32.101:8091, MachineId=2149, Location=machine:dev1ssapp3,process:7803, Role=ApacheCatalinaStartupBootstrap)
Member(Id=14, Timestamp=2012-07-30 15:35:09.811, Address=10.34.32.106:8088, MachineId=2154, Location=machine:dev1sxapp1,process:5186, Role=ApacheCatalinaStartupBootstrap)
Member(Id=15, Timestamp=2012-07-30 16:02:34.096, Address=10.34.32.106:8091, MachineId=2154, Location=machine:dev1sxapp1,process:2691, Role=ApacheCatalinaStartupBootstrap)
Member(Id=16, Timestamp=2012-07-30 16:08:41.885, Address=10.34.32.107:8091, MachineId=2155, Location=machine:dev1sxapp2,process:15992, Role=ApacheCatalinaStartupBootstrap)
Member(Id=21, Timestamp=2012-07-30 21:58:56.669, Address=10.34.32.106:8089, MachineId=2154, Location=machine:dev1sxapp1,process:28689, Role=ApacheCatalinaStartupBootstrap)
Member(Id=22, Timestamp=2012-07-30 21:58:58.29, Address=10.34.32.107:8090, MachineId=2155, Location=machine:dev1sxapp2,process:15491, Role=ApacheCatalinaStartupBootstrap)
Member(Id=23, Timestamp=2012-07-30 22:00:21.648, Address=10.34.32.106:8090, MachineId=2154, Location=machine:dev1sxapp1,process:556, Role=ApacheCatalinaStartupBootstrap)
MemberId/ServiceVersion/ServiceJoined/ServiceLeaving
1/3.5/Fri Jul 27 10:46:51 EDT 2012/false,
2/3.5/Fri Jul 27 10:47:12 EDT 2012/false,
3/3.5/Fri Jul 27 10:48:02 EDT 2012/false,
4/3.5/Fri Jul 27 10:48:04 EDT 2012/false,
8/3.5/Mon Jul 30 14:27:07 EDT 2012/false,
9/3.5/Mon Jul 30 22:00:28 EDT 2012/false,
10/3.5/Mon Jul 30 14:34:27 EDT 2012/false,
11/3.5/Mon Jul 30 22:00:41 EDT 2012/false,
12/3.5/Mon Jul 30 22:00:43 EDT 2012/false,
14/3.5/Mon Jul 30 15:35:09 EDT 2012/false,
15/3.5/Mon Jul 30 16:02:34 EDT 2012/false,
16/3.5/Mon Jul 30 16:08:41 EDT 2012/false,
21/3.5/Mon Jul 30 21:58:56 EDT 2012/false,
22/3.5/Mon Jul 30 21:58:58 EDT 2012/false,
23/3.5/Mon Jul 30 22:00:21 EDT 2012/false
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:38)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:366)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:7)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:27)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:1011)
at com.tangosol.coherence.hibernate.CoherenceCacheProvider.start(CoherenceCacheProvider.java:73)
at org.hibernate.cache.impl.bridge.RegionFactoryCacheProviderBridge.start(RegionFactoryCacheProviderBridge.java:72)
Looking at member 5's log and I found it was being bounced at that time but somehow it failed to stop the coherence thread and didn't send out departure event to the cluster until was requested by other members.
SEVERE: The web application [riding-services] appears to have started a thread named Cluster but has failed to stop it. This is very likely to create a memory leak.
Questions:
1. Seems that this issue only happens when one server starts while another server is shut down at the same time range and both happen to be connected with each other for distributed caching. How can I modify the script to retry during startup when the first time it timed out? Or maybe modify the configuration to use a longer timeout value?
2. Is it possible to detect the unavailability of certain member quicker? Now seems 30 seconds or more.
Thanks in advance, -
Coherence Cluster Errors- Need your help to solve
Hi,
We had this error recently in QA and these servers are not new servers. These servers were running from some time and in good condition.
We had a below error happened suddently and cuased servers outage for some time.
After restarted all the servers, this issue has gone.
We are trying to understand the root cause to avoid this issue in future and need expertise in this forum for that.
Brief summary of issue
1. We had performed multicaste testing on the coherence cluster IP and port and all the communication is good.
2. Issues started with error of Unable to refresh sockets:
Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: Unable to refresh sockets: [UnicastUdpSocket{State=STATE_OPEN, address:port=1.1.1.85:8088}, MulticastUdpSocket{State=STATE_OPEN, address:port=239.3.1.17:35122, InterfaceAddress=10.137.3.85, TimeToLive=1}, TcpSocketAccepter{State=STATE_OPEN, ServerSocket=1.1.1.85:8088}]; last failed socket: MulticastUdpSocket{State=STATE_OPEN, address:port=239.3.1.17:35122, InterfaceAddress=10.137.3.85, TimeToLive=1}
at com.tangosol.coherence.component.net.Cluster$SocketManager.refreshSockets(Cluster.CDB:91)
at com.tangosol.coherence.component.net.Cluster$SocketManager$MulticastUdpSocket.onInterruptedIOException(Cluster.CDB:9)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:33)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: Receive timed out
3. After that, I noticed copule of errors like
Restarting Service: DistributedCache validatePolls: This service timed-out due to unanswered handshake request. Manual intervention is required to stop the members that have not responded to this Poll
4. Continously logging errors like : Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/
5. After that noticed,
Service DistributedCache: received ServiceConfigSync containing 272 entries
2013-10-26 08:26:43,241 -0700 level=ERROR class="STDERR" 2013-10-26 08:26:43.241/76.243 Oracle Coherence GE 3.5.1/461 <Error> (thread=main, member=1): Error while starting service "DistributedCache": com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=2, Name=DistributedCache, Type=DistributedCache
MemberSet=ServiceMemberSet(
OldestMember=Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=10.137.3.49:8088, MachineId=32817, Location=site:test.test.net,machine:test30b,process:3870)
ActualMemberSet=MemberSet(Size=3, BitSetCount=2
Member(Id=1, Timestamp=2013-10-26 08:26:12.289, Address=1.1.1.85:8088, MachineId=32853, Location=site:test.test.net,machine:test304,process:6207, Role=JavaLangThread)
Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1.49:8088, MachineId=32817, Location=site:test.test.net,machine:test30b,process:3870)
Member(Id=5, Timestamp=2013-10-26 08:26:29.871, Address=1.1.1.86:8088, MachineId=32854, Location=site:test.test.net,machine:test305,process:3988)
MemberId/ServiceVersion/ServiceJoined/ServiceLeaving
1/3.5/Sat Oct 26 08:26:13 PDT 2013/false,
3/3.5/Sat Oct 26 05:16:47 PDT 2013/false,
5/3.5/Sat Oct 26 08:26:30 PDT 2013/false
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
Your Help is highly appreciated !!!!
Detailed Server Error Log:
2013-10-26 00:15:13,280 -0700 level=ERROR class="STDERR"
2013-10-26 00:15:13.279/2079180.072 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 2642 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread); 34 packets rescheduled, PauseRate=0.0010, Threshold=222
2013-10-26 00:15:15,508 -0700 level=ERROR class="STDERR"
2013-10-26 00:15:15.508/2079182.301 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 4875 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2013-10-08 22:00:17.258, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Role=JavaLangThread); 47 packets rescheduled, PauseRate=3.0E-4, Threshold=1438
2013-10-26 01:15:29,028 -0700 level=ERROR class="STDERR"
2013-10-26 01:15:29.018/2082795.811 Oracle Coherence GE 3.5.1/461 <Info> (thread=PacketListenerN, member=4): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2013-10-26 01:15:29,036 -0700 level=ERROR class="STDERR"
2013-10-26 01:15:29.036/2082795.829 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 13068 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2013-10-08 22:00:17.258, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Role=JavaLangThread); 86 packets rescheduled, PauseRate=4.0E-4, Threshold=1438
2013-10-26 01:15:29,037 -0700 level=ERROR class="STDERR"
2013-10-26 01:15:29.036/2082795.829 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 13069 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread); 84 packets rescheduled, PauseRate=0.0010, Threshold=269
2013-10-26 01:31:44,494 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 02:15:34,907 -0700 level=ERROR class="STDERR"
2013-10-26 02:15:34.906/2086401.699 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 6476 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread); 24 packets rescheduled, PauseRate=0.0011, Threshold=313
2013-10-26 02:43:52,199 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 03:00:55,493 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 03:15:41,144 -0700 level=ERROR class="STDERR"
2013-10-26 03:15:41.144/2090007.937 Oracle Coherence GE 3.5.1/461 <D5> (thread=PacketPublisher, member=4): Experienced a 202 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2013-10-08 22:00:17.258, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Role=JavaLangThread); 25 packets rescheduled, PauseRate=4.0E-4, Threshold=1509
2013-10-26 03:15:41,592 -0700 level=ERROR class="STDERR"
2013-10-26 03:15:41.592/2090008.385 Oracle Coherence GE 3.5.1/461 <D5> (thread=PacketPublisher, member=4): Experienced a 371 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread); 41 packets rescheduled, PauseRate=0.0010, Threshold=290
2013-10-26 03:31:38,099 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 04:15:47,869 -0700 level=ERROR class="STDERR"
2013-10-26 04:15:47.869/2093614.662 Oracle Coherence GE 3.5.1/461 <D5> (thread=PacketPublisher, member=4): Experienced a 850 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2013-10-08 22:00:17.258, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Role=JavaLangThread); 52 packets rescheduled, PauseRate=4.0E-4, Threshold=1509
2013-10-26 04:16:00,192 -0700 level=ERROR class="STDERR"
2013-10-26 04:16:00.182/2093626.975 Oracle Coherence GE 3.5.1/461 <Info> (thread=PacketListenerN, member=4): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2013-10-26 04:16:00,199 -0700 level=ERROR class="STDERR"
2013-10-26 04:16:00.199/2093626.992 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 13180 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread); 126 packets rescheduled, PauseRate=0.0011, Threshold=424
2013-10-26 04:16:01,897 -0700 level=ERROR class="STDERR"
2013-10-26 04:16:01.897/2093628.690 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=4): Experienced a 1503 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2013-10-08 22:00:17.258, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Role=JavaLangThread); 173 packets rescheduled, PauseRate=4.0E-4, Threshold=1509
2013-10-26 04:26:54,424 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 04:51:52,096 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:02:52,292 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:16:06,076 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.075/2097232.868 Oracle Coherence GE 3.5.1/461 <Error> (thread=PacketListenerN, member=4):
Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: Unable to refresh sockets: [UnicastUdpSocket{State=STATE_OPEN, address:port=1.1.1..85:8088}, MulticastUdpSocket{State=STATE_OPEN, address:port=239.3.1.17:35122, InterfaceAddress=1.1.1..85, TimeToLive=1}, TcpSocketAccepter{State=STATE_OPEN, ServerSocket=1.1.1..85:8088}]; last failed socket: MulticastUdpSocket{State=STATE_OPEN, address:port=239.3.1.17:35122, InterfaceAddress=1.1.1..85, TimeToLive=1}
at com.tangosol.coherence.component.net.Cluster$SocketManager.refreshSockets(Cluster.CDB:91)
at com.tangosol.coherence.component.net.Cluster$SocketManager$MulticastUdpSocket.onInterruptedIOException(Cluster.CDB:9)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:33)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: Receive timed out
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
at java.net.DatagramSocket.receive(DatagramSocket.java:725)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
2013-10-26 05:16:06,080 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.080/2097232.873 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=4): Service Cluster left the cluster
2013-10-26 05:16:06,105 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.105/2097232.898 Oracle Coherence GE 3.5.1/461 <D5> (thread=Invocation:Management, member=4): Service Management left the cluster
2013-10-26 05:16:06,105 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.105/2097232.898 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-180, member=4): Restarting NamedCache: test234aaaapeu-cache
2013-10-26 05:16:06,105 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.105/2097232.898 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-180, member=4): Restarting Service: DistributedCache
2013-10-26 05:16:06,110 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.106/2097232.899 Oracle Coherence GE 3.5.1/461 <Error> (thread=DistributedCache, member=4):
validatePolls: This service timed-out due to unanswered handshake request. Manual intervention is required to stop the members that have not responded to this Poll
PollId=24209529, active
InitTimeMillis=1382789736843
Service=DistributedCache (2)
RespondedMemberSet=[]
LeftMemberSet=[]
RemainingMemberSet=[3]
Request=Message "LockRequest"
{test.test.net
FromMember=Member(Id=4, Timestamp=2013-10-24 15:16:09.067, Address=1.1.1..85:8088, MachineId=32853, Location=site:test.test.net,machine:testabc304,process:4000)
FromMessageId=38338332
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=12
ToPollId=0
Poll=null
Packets
Service=DistributedCache{Name=DistributedCache, State=(SERVICE_STOPPED), Not initialized}
ToMemberSet=MemberSet(Size=1, BitSetCount=1
Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread)
NotifySent=false
null
WaitTimeout=1382789776739, LeaseExpiration=9223372036854775807
2013-10-26 05:16:06,110 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.109/2097232.902 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=4): Service DistributedCache left the cluster
2013-10-26 05:16:06,117 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.117/2097232.910 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-180, member=n/a): Restarting cluster
2013-10-26 05:16:06,198 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:06.198/2097232.991 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2013-10-26 05:16:07,410 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.410/2097234.203 Oracle Coherence GE 3.5.1/461 <Info> (thread=Cluster, member=n/a): Created a new cluster "cluster:0x27CB" with Member(Id=1, Timestamp=2013-10-26 05:16:06.128, Address=1.1.1..85:8088, MachineId=32853, Location=site:test.test.net,machine:testabc304,process:4000, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4) UID=0x0A89035500000141F4B15BF080551F98
2013-10-26 05:16:07,436 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.436/2097234.229 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-180, member=1): Restarting Service: Management
2013-10-26 05:16:07,450 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.450/2097234.243 Oracle Coherence GE 3.5.1/461 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
2013-10-26 05:16:07,474 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.474/2097234.267 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 1
2013-10-26 05:16:07,491 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.491/2097234.284 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-183, member=1): Restarting NamedCache: test234aaaaficustomer-cache
2013-10-26 05:16:07,514 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.514/2097234.307 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-38, member=1): Restarting NamedCache: test234aaaaaccount-no-export-cache
2013-10-26 05:16:07,529 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.529/2097234.322 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-136, member=1): Restarting NamedCache: test234aaaausrsum-cache
2013-10-26 05:16:07,546 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.545/2097234.338 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-9, member=1): Restarting NamedCache: test234aaaafi-v2-cache
2013-10-26 05:16:07,569 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.567/2097234.360 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-59, member=1): Restarting NamedCache: test234aaaaaccount-v2-cache
2013-10-26 05:16:07,748 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.748/2097234.541 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-28, member=1): Restarting NamedCache: test234aaaafi-cache
2013-10-26 05:16:07,816 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:07.816/2097234.609 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-133, member=1): Restarting NamedCache: test234aaaahistory-v2-cache
2013-10-26 05:16:09,154 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.154/2097235.947 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-134, member=1): Restarting NamedCache: test234aaaaaccount-cache
2013-10-26 05:16:09,169 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.169/2097235.962 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-134, member=1): Restarting NamedCache: test234aaaahistory-cache
2013-10-26 05:16:09,444 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.444/2097236.237 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2013-10-26 05:16:09.259, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988) joined Cluster with senior member 1
2013-10-26 05:16:09,539 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.539/2097236.332 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 2 joined Service Management with senior member 1
2013-10-26 05:16:09,580 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.579/2097236.372 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 2 joined Service DistributedCache with senior member 1
2013-10-26 05:16:09,599 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.599/2097236.392 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Service DistributedCache: sending ServiceConfigSync containing 268 entries to Member 2
2013-10-26 05:16:09,681 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.681/2097236.474 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 1> Transferring 128 out of 257 vulnerable partitions to member 2 requesting 128
2013-10-26 05:16:09,892 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.881/2097236.674 Oracle Coherence GE 3.5.1/461 <D4> (thread=DistributedCache, member=1): 1> Transferring 129 out of 129 partitions to a machine-safe backup 1 at member 2 (under 129)
2013-10-26 05:16:09,901 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:09.901/2097236.694 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 388KB of backup[1] for PartitionSet{128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256} to member 2
2013-10-26 05:16:10,415 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:10.415/2097237.208 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): TcpRing: connecting to member 2 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..86,port=8088,localport=37005]}
2013-10-26 05:16:10,657 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:10.657/2097237.450 Oracle Coherence GE 3.5.1/461 <Warning> (thread=Cluster, member=1): Received panic from junior member Member(Id=2, Timestamp=2013-10-26 05:16:09.259, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988) caused by Member(Id=3, Timestamp=2013-10-01 22:43:27.913, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870, Role=JavaLangThread)
2013-10-26 05:16:11,592 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:11.592/2097238.385 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32822,localport=8088]}
2013-10-26 05:16:13,568 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:13.568/2097240.361 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-52, member=1): Restarting NamedCache: test234aaaauserData-cache
2013-10-26 05:16:13,596 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:13.596/2097240.389 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32823,localport=8088]}
2013-10-26 05:16:14,937 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:14.937/2097241.730 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-52, member=1): Restarting NamedCache: test234aaaacheckimage-cache
2013-10-26 05:16:15,600 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:15.600/2097242.393 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32824,localport=8088]}
2013-10-26 05:16:17,602 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:17.602/2097244.395 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32825,localport=8088]}
2013-10-26 05:16:19,605 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:19.605/2097246.398 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32828,localport=8088]}
2013-10-26 05:16:21,609 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:21.609/2097248.402 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32829,localport=8088]}
2013-10-26 05:16:23,611 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:23.611/2097250.404 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32830,localport=8088]}
2013-10-26 05:16:25,616 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:25.616/2097252.409 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32831,localport=8088]}
2013-10-26 05:16:27,619 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:27.619/2097254.412 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32832,localport=8088]}
2013-10-26 05:16:29,621 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:29.621/2097256.414 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32833,localport=8088]}
2013-10-26 05:16:31,626 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:31.626/2097258.419 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32834,localport=8088]}
2013-10-26 05:16:33,631 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:33.631/2097260.424 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32835,localport=8088]}
2013-10-26 05:16:35,632 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:35.632/2097262.425 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32836,localport=8088]}
2013-10-26 05:16:37,636 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:37.635/2097264.428 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32837,localport=8088]}
2013-10-26 05:16:39,641 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:39.640/2097266.433 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32838,localport=8088]}
2013-10-26 05:16:41,643 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:41.643/2097268.436 Oracle Coherence GE 3.5.1/461 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32841,localport=8088]}
2013-10-26 05:16:47,329 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:47.329/2097274.122 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870) joined Cluster with senior member 1
2013-10-26 05:16:47,425 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:47.425/2097274.218 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 3 joined Service Management with senior member 1
2013-10-26 05:16:47,477 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:47.476/2097274.269 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 3 joined Service DistributedCache with senior member 1
2013-10-26 05:16:47,501 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:47.500/2097274.294 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Service DistributedCache: sending ServiceConfigSync containing 270 entries to Member 3
2013-10-26 05:16:47,548 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:47.548/2097274.341 Oracle Coherence GE 3.5.1/461 <D5> (thread=TcpRingListener, member=1): TcpRing: connecting to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=32846,localport=8088]}
2013-10-26 05:16:48,454 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:48.453/2097275.246 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 43 out of 129 primary partitions to member 3 requesting 43
2013-10-26 05:16:48,709 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:48.709/2097275.502 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 39 out of 125 primary partitions to member 3 requesting 39
2013-10-26 05:16:48,885 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:48.884/2097275.677 Oracle Coherence GE 3.5.1/461 <D5> (thread=http-0.0.0.0-8080-210, member=1): Repeating QueryRequest due to the re-distribution of PartitionSet{132, 133, 134, 135, 136, 137, 138, 139, 140, 141}
2013-10-26 05:16:50,850 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:50.848/2097277.641 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 29 out of 115 primary partitions to member 3 requesting 29
2013-10-26 05:16:50,968 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:50.968/2097277.761 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 21 out of 107 primary partitions to member 3 requesting 21
2013-10-26 05:16:51,097 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.097/2097277.890 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 14 out of 100 primary partitions to member 3 requesting 14
2013-10-26 05:16:51,218 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.218/2097278.011 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 6 out of 92 primary partitions to member 3 requesting 6
2013-10-26 05:16:51,340 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.340/2097278.133 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): 2> Transferring 1 out of 87 primary partitions to member 3 requesting 1
2013-10-26 05:16:51,352 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.352/2097278.145 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 540KB of backup[1] for PartitionSet{171, 172, 173, 174, 175, 176, 177} to member 3
2013-10-26 05:16:51,465 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.464/2097278.257 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 575KB of backup[1] for PartitionSet{178, 179, 180, 181, 182, 183} to member 3
2013-10-26 05:16:51,569 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.569/2097278.362 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 537KB of backup[1] for PartitionSet{184, 185, 186, 187} to member 3
2013-10-26 05:16:51,688 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.688/2097278.481 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 553KB of backup[1] for PartitionSet{188, 189, 190, 191, 192, 193, 194, 195, 196} to member 3
2013-10-26 05:16:51,817 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.817/2097278.610 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 526KB of backup[1] for PartitionSet{197, 198, 199, 200, 201, 202} to member 3
2013-10-26 05:16:51,928 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:51.928/2097278.721 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 768KB of backup[1] for PartitionSet{203, 204, 205, 206, 207, 208, 209} to member 3
2013-10-26 05:16:52,040 -0700 level=ERROR class="STDERR"
2013-10-26 05:16:52.039/2097278.832 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Transferring 198KB of backup[1] for PartitionSet{210, 211, 212, 213} to member 3
2013-10-26 05:19:06,157 -0700 level=ERROR class="STDERR"
2013-10-26 05:19:06.157/2097412.950 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-63, member=1): Restarting NamedCache: throttleData-cache
2013-10-26 05:22:15,094 -0700 level=ERROR class="STDERR"
2013-10-26 05:22:15.094/2097601.887 Oracle Coherence GE 3.5.1/461 <Info> (thread=http-0.0.0.0-8080-136, member=1): Restarting NamedCache: test234aaaadepositslipimage-cache
2013-10-26 05:22:17,183 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:28:49,617 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:29:39,729 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:33:37,607 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 05:39:33,872 -0700 level=INFO class="STDOUT"
WARN getResponseBody, Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
2013-10-26 06:49:30,617 -0700 level=ERROR class="STDERR"
2013-10-26 06:49:30.617/2102837.410 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=1): Experienced a 6378 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-26 05:16:09.259, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988); 56 packets rescheduled, PauseRate=0.0011, Threshold=1976
2013-10-26 07:39:18,855 -0700 level=ERROR class="STDERR"
2013-10-26 07:39:18.854/2105825.647 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=1): Experienced a 7318 ms communication delay (probable remote GC) with Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870); 68 packets rescheduled, PauseRate=8.0E-4, Threshold=497
2013-10-26 07:49:37,510 -0700 level=ERROR class="STDERR"
2013-10-26 07:49:37.510/2106444.303 Oracle Coherence GE 3.5.1/461 <Warning> (thread=PacketPublisher, member=1): Experienced a 6653 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-26 05:16:09.259, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988); 69 packets rescheduled, PauseRate=0.0014, Threshold=1785
Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
2013-10-26 08:26:11,291 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:11.291/44.293 Oracle Coherence GE 3.5.1/461 <Info> (thread=main, member=n/a): Loaded cache configuration from "file:/usr/local/whp-jboss-web-5/server/default/env/test234aaaacoherence-cache-config.xml"
2013-10-26 08:26:12,263 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.263/45.265 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2013-10-26 08:26:12,477 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.477/45.479 Oracle Coherence GE 3.5.1/461 <Info> (thread=Cluster, member=n/a): This Member(Id=1, Timestamp=2013-10-26 08:26:12.289, Address=1.1.1..85:8088, MachineId=32853, Location=site:test.test.net,machine:testabc304,process:6207, Role=JavaLangThread, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4) joined cluster "cluster:0x27CB" with senior Member(Id=2, Timestamp=2013-10-26 05:16:09.259, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988, Edition=Grid Edition, Mode=Development, CpuCount=4, SocketCount=4)
2013-10-26 08:26:12,501 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.501/45.503 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870) joined Cluster with senior member 2
2013-10-26 08:26:12,507 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.506/45.508 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 2 joined Service Management with senior member 2
2013-10-26 08:26:12,507 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.507/45.509 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 2 joined Service DistributedCache with senior member 2
2013-10-26 08:26:12,520 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.520/45.522 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 3 joined Service Management with senior member 2
2013-10-26 08:26:12,520 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.520/45.522 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Member 3 joined Service DistributedCache with senior member 2
2013-10-26 08:26:12,639 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.639/45.641 Oracle Coherence GE 3.5.1/461 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 2
2013-10-26 08:26:12,700 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:12.700/45.702 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): TcpRing: connecting to member 3 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..49,port=8088,localport=52891]}
2013-10-26 08:26:13,191 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:13.190/46.193 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 2
2013-10-26 08:26:14,538 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:14.538/47.540 Oracle Coherence GE 3.5.1/461 <D5> (thread=TcpRingListener, member=1): TcpRing: connecting to member 2 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..86,port=40281,localport=8088]}
2013-10-26 08:26:29,695 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:29.694/62.696 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): TcpRing: disconnected from member 2 due to a kill request
2013-10-26 08:26:29,695 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:29.694/62.696 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 2 left service Management with senior member 3
2013-10-26 08:26:29,695 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:29.694/62.696 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 2 left service DistributedCache with senior member 3
2013-10-26 08:26:29,696 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:29.696/62.698 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2013-10-26 08:26:29.694, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988) left Cluster with senior member 3
2013-10-26 08:26:30,069 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:30.069/63.071 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member(Id=5, Timestamp=2013-10-26 08:26:29.871, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988) joined Cluster with senior member 3
2013-10-26 08:26:30,271 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:30.271/63.273 Oracle Coherence GE 3.5.1/461 <D5> (thread=TcpRingListener, member=1): TcpRing: connecting to member 5 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/1.1.1..86,port=40285,localport=8088]}
2013-10-26 08:26:30,272 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:30.272/63.274 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 5 joined Service Management with senior member 3
2013-10-26 08:26:30,443 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:30.443/63.445 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=1): Member 5 joined Service DistributedCache with senior member 3
2013-10-26 08:26:38,739 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:38.738/71.740 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=1): Service DistributedCache: received ServiceConfigSync containing 272 entries
2013-10-26 08:26:43,241 -0700 level=ERROR class="STDERR"
2013-10-26 08:26:43.241/76.243 Oracle Coherence GE 3.5.1/461 <Error> (thread=main, member=1): Error while starting service "DistributedCache": com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=2, Name=DistributedCache, Type=DistributedCache
MemberSet=ServiceMemberSet(
OldestMember=Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870)
ActualMemberSet=MemberSet(Size=3, BitSetCount=2
Member(Id=1, Timestamp=2013-10-26 08:26:12.289, Address=1.1.1..85:8088, MachineId=32853, Location=site:test.test.net,machine:testabc304,process:6207, Role=JavaLangThread)
Member(Id=3, Timestamp=2013-10-26 05:16:47.128, Address=1.1.1..49:8088, MachineId=32817, Location=site:test.test.net,machine:testabc30b,process:3870)
Member(Id=5, Timestamp=2013-10-26 08:26:29.871, Address=1.1.1..86:8088, MachineId=32854, Location=site:test.test.net,machine:testabc305,process:3988)
MemberId/ServiceVersion/ServiceJoined/ServiceLeaving
1/3.5/Sat Oct 26 08:26:13 PDT 2013/false,
3/3.5/Sat Oct 26 05:16:47 PDT 2013/false,
5/3.5/Sat Oct 26 08:26:30 PDT 2013/false
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:38)
at com.tangosol.coherence.component.util.SafeService.startService(SafeService.CDB:28)
at com.tangosol.coherence.component.util.safeService.SafeCacheService.startService(SafeCacheService.CDB:5)
at com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:27)
at com.tangosol.coherence.component.util.SafeService.start(SafeService.CDB:14)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:973)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:842)
at com.tangosol.net.DefaultConfigurableCacheFactory.configureCache(DefaultConfigurableCacheFactory.java:1053)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:290)
at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:747)
at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:724Hi
The common causes of communication delays and packet timeouts are excessive GC pauses, high CPU usage, and swapping.
Each of these occurrences may disrupt the Coherence packet processing threads, thus preventing the processing and acknowledgment of packets from other cluster members.
1 check GC performance , see process memory consumption and GC logs.
2 check cpu , vmstat , top command.
3 check swap , vmstat command.
see Oracle Support Doc ID 1110544.1
Although communication delays and packet timeouts can be caused by network related issue.
check performance network :
Performing a Datagram Test for Network Performance - Coherence 3.5 User Guide - Oracle Coherence Knowledge Base
regards,
Leo_TA -
Cluster with shared domain folder using OCFS2 and the domain /tmp folder
I've setup a cluster with two machines and one managed server on each machine. One of the machines also hosts the admin server.
Initially, both managed servers were running on the same machine. Everything worked well. When I move to having two machines, I didn't want to have to independently maintain the domain folder on each machine.
So I decided to setup a shared drive between the machines and use OCFS2 to share the domain folder between the two machines.
Everything started up fine, but when I deployed software to the server I would frequently (but not always) get this error:
[wldeploy] Target state: deploy failed on Cluster cluster-01
[wldeploy] weblogic.management.DeploymentException: [Deployer:149189]Attempt to operate 'deploy' on null BasicDeploymentMBean for deployment mylibrary#[email protected]. Operation can not be performed until server is restarted.
After looking around a bit on google I saw this issue:
http://download.oracle.com/docs/cd/E11035_01/wls100/issues/known_resolved.html#CR279281
which says
Some OS and NFS combinations result in deployment failures or configuration updates with an exception like:
weblogic.management.DeploymentException: Attempt to operate 'distribute' on null BasicDeploymentMBean
Workaround or Solution:
* Run statd() and lockd() processes on every NFS client that accesses a remote NFS volume.
* If multiple servers that share the same domain root are started with different user Ids of same group, set the correct "umask" for the server processes so that a file created by one server can be opened for read/write by other servers without security exceptions.
Unfortunately, I wasn't running NFS, so the workaround didn't help me.
Through dump luck, I tried this:
I replaced the domainroot/tmp folder with a symbolic link to a folder on each machine's local filesystem. Following that change my deployments work everytime.
Does anyone know why this change worked? What is the purpose of the /tmp folder in a domain folder?
Are there any switches I could turn on to get more information about the exact error that causes the deployment mbean to come back null?It may not be the same situation, but I had this "deployer:149189" error and it turned out to be mixing WLS 10.0 and 10.3 installations together in the same install directory and not being scrupulous about environment setup. I think the problem was caused by accidentally running a 10.0 domain with Java 1.6.
Eoin. -
Persistent Store Problems for MYSQL Enhanced Cluster With OpenMQ 4.4
I am trying to implement an enhanced cluster with failover. I have edited the config files for each broker instance for a persistent store. I have appended the following to each of the config.properties files:
imq.brokerid=myclusterinstanceINSTANCE1 # I substitute INSTANCE2 for INSTANCE1 for broker #2
imq.persist.store=jdbc
imq.persist.jdbc.dbVendor=mysql
imq.persist.jdbc.mysql.property.url=jdbc:mysql://xxx.xxx.xxx.xx:3306/test
imq.persist.jdbc.mysql.user=user1
imq.persist.jdbc.mysql.needpassword=true
imq.persist.jdbc.mysql.password=mypass
imq.cluster.ha=true
imq.cluster.clusterid=mycluster
imq.cluster.brokerlist=xxx.xxx.xxx.x:37676,yyy.yyy.yyy.y:37676
I then create the persistence storage with "imqdbmgr create tbl". When I view the data in the tables it creates, I have one row. Under Store_Version, I have 410. Under LOCK_ID, it has NULL. When I go to start the brokers with imqbrokerd, I get the following error:
ERROR [B3198]: Error initializing cluster manager:
com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4239]: Failed to load persistent store version from database table MQVER41Cmycluster
at com.sun.messaging.jmq.jmsserver.persist.jdbc.VersionDAOImpl.getStoreVersion(VersionDAOImpl.java:310)
at com.sun.messaging.jmq.jmsserver.persist.jdbc.DBTool.updateStoreVersion410IfNecessary(DBTool.java:350)
at com.sun.messaging.jmq.jmsserver.persist.jdbc.JDBCStore.checkStore(JDBCStore.java:3599)
at com.sun.messaging.jmq.jmsserver.persist.jdbc.JDBCStore.<init>(JDBCStore.java:127)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at com.sun.messaging.jmq.jmsserver.persist.StoreManager.getStore(StoreManager.java:157)
at com.sun.messaging.jmq.jmsserver.Globals.getStore(Globals.java:967)
at com.sun.messaging.jmq.jmsserver.cluster.ha.HAClusterManagerImpl.initialize(HAClusterManagerImpl.java:181)
at com.sun.messaging.jmq.jmsserver.Globals.initClusterManager(Globals.java:903)
at com.sun.messaging.jmq.jmsserver.Broker._start(Broker.java:777)
at com.sun.messaging.jmq.jmsserver.Broker.start(Broker.java:410)
at com.sun.messaging.jmq.jmsserver.Broker.main(Broker.java:1971)
Caused by: java.lang.NullPointerException
at com.mysql.jdbc.ResultSetImpl.findColumn(ResultSetImpl.java:1103)
at com.mysql.jdbc.ResultSetImpl.getInt(ResultSetImpl.java:2777)
at com.sun.messaging.jmq.jmsserver.persist.jdbc.VersionDAOImpl.getStoreVersion(VersionDAOImpl.java:298)
... 16 more
I believe this error is attributed to the NULL value under LOCK_ID. I think that the value under LOCK_ID should be the name of the broker from the config file (even though I specified them in the config files). Any ideas?? THANKS!Just some pointers -- maybe this will be of use:
If you haven't already read it, please take a look at the [ MySQL setup guide|https://mq.dev.java.net/OpenMQ_MySQLCluster_Setup_Guide.html] .
We recommend using NDB Data-store of MySQL Cluster, though this isn't an absolute requirement. Due to some issues we have found with earlier versions, we recommend using MySQL Cluster, 7.0.9 or better (the current version is 7.0.16, or 7.1.5). Either of these would contain Connector/J.
I'd also recommend using the latest version -- MQ 4.4update2 (just in case you happen to have an older copy). There were many minor improvements in the integration with MySQL from the original 4.4 release, to update 2. This is linked at the MQ download page: [https://mq.dev.java.net/downloads.html] -
We have a MSSQL 2008R2 Enterprise edition with a two node active passive fail-over cluster running on 2008R2 windows cluster with out any issues,
Now my question is can we add one more MSSSQL cluster instance for the same setup with out disturbing the existing one ?
Also give thoughts on load sharing as the second node is mostly ideal now except fail-over scenarios,
Why we go for this situation is because of the collation setting which can be set only one per instance(Database collation setting change not working), we need a different default collation for the new setuphi,
>>Now my question is can we add one more MSSSQL cluster instance for the same setup with out disturbing the existing one ?
Yes it is possible .You need to add new drives as cluster aware and install SQL server and put data and log files on thse drives.YOu would need to create named instance of SQL server and need to create different resource group.Both old installation and new
onw would work separately.
>>Also give thoughts on load sharing as the second node is mostly ideal now except fail-over scenarios,
Good point indeed.You are about to create Multi instance cluster and should plan for scenario where one node is down and other node is handling load for both instances.Memory and CPU should be enough to handle the load.
>>Why we go for this situation is because of the collation setting which can be set only one per instance(Database collation setting change not working), we need a different default collation for the new setup .
Just for collation if you are installing new instance seems little wierd to me.You can manage collation at column ,database and at server level.
http://technet.microsoft.com/en-us/library/aa174903(v=sql.80).aspx
Please mark this reply as the answer or vote as helpful, as appropriate, to make it useful for other readers
Maybe you are looking for
-
Error Message: Illegal memory access. [54] in jrockit/vm/Reflects$IClass
Hi, We ve got an Illegal memory access during stress tests. Is it a known crash? Let us know other info are needed. Thanks in advance, Marian Here is the jrockit.<pid>.dump file: ===== BEGIN DUMP ======================================================
-
What is the highest OS a second generation ipod touch can go up to?
What is the highest OS a second generation ipod touch can go up to?
-
Mac G5 Flash CS3 Hi, I'm creating a movie with a swift flies quickly across screen, but in playback it always appears as if there are two or three swifts flying together almost on top of one another, a sort of stuttering effect in the playback. I've
-
Hi! Has anyone had problems with their email notifications? I am not getting notifications for my work email. It had been working fine then suddenly it wasn't. I went to the VZW store to get it looked into. They took out the battery and sim card fo
-
Problem with Audigy 2 ZS and Microph
hi! i use skype to communicate with my friends. in sound mixer i chose 'microphone' for recording. this works like a charm. BUT: as soon as i start a game (i.e. World of Warcraft), the recording device is automatically changed to 'what you hear'. whe