StopRunning ClusterService
I stopped by WebLogic Server, but the script failed, so I had to kill -9 the startWeblogic.s process and the java process. Now I cannot restart the server. Whenever I try, I get the following error:
<Jan 6, 2015 12:17:48 PM PST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY.>
<Jan 6, 2015 12:17:48 PM PST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
<Jan 6, 2015 12:17:48 PM PST> <Notice> <Log Management> <BEA-170032> <The Server has detected ODL configuration for java.util.logging. The PlatformLoggerLevels attribute configuration on the LogMBean will be ignored.>
<Error>: Failed to write a log message
<Jan 6, 2015 12:18:07 PM PST> <Error> <com.oracle.coherence> <BEA-000000> <2015-01-06 12:18:07.572/40.183 Oracle Coherence GE 12.1.3.0.0 <Error> (thread=Cluster, member=1): StopRunning ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=12.1.3, OldestMemberId=1} due to unhandled exception: >
Any suggestions for how to resolve this?
Thank you!
Dan
Thanks for the advice. Unfortunately, my coherence is still failing somehow. It is strange because my Admin server was working up until yesterday. In any event, no matter how many domains I create, I still get the following error:
<Logger@1032327427 12.1.3.0.0> <<anonymous>> <> <> <1420656852991> <BEA-000000> <2015-01-07 10:54:12.991/60.973 Oracle Coherence GE 12.1.3.0.0 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2015-01-07 10:53:42.419, Address=<ipaddress>:7202, MachineId=9428, Location=site:,machine:hostname,process:26602,member:AdminServer, Role=WeblogicServer) has been attempting to join the cluster using WKA list [/<ipaddress>:7201] for 30 seconds without success; this could indicate a mis-configured WKA, or it may simply be the result of a busy cluster or active failover.>
Thanks again for your hlep,
Daniel
Similar Messages
-
Dealing with hard timeout of guarded service
Hi, I'm investigating the behavior the cause of and subsequent behavior after a hard timeout of a guarded service. In my experience, the grid members are unable to recover properly. I am trying to figure out whether there is something in our configuration that may be aggravating the situation, and also see whether I might be able to improve the behavior of our client code.
What I have done is purposely lower the service guardian's timeout, to about 25 seconds, so that a certain EntryProcessor will always time-out. The behavior I see after the timeout is very similar to the behavior we see when the problem crops up in the real world. I know that I can raise the timeout or use the PriorityTask API, but even if I do that, we may run into timeouts due to deadlock. If that happens, I want to know what I can expect the grid to do.
Here's what I'm seeing:
I am running 2 storage-enabled members with backup-count=0. I am running my "client" as a non-storage-enabled member.
First, the client kicks off the EntryProcessor:
Map<Object, Integer> readProcessorResult = requestCache.invokeAll(AlwaysFilter.INSTANCE, processor);
The two members begin logging successful executions of the EntryProcessor. Then, one of the members happens to run it against an entry that takes longer than the service guardian's timeout. We get a soft and hard timeout:
[ERROR] Coherence: 2012-01-26 10:56:20.103/333.297Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=1): Detected soft timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:jdw-read-request-service} Service=PartitionedCache{Name=jdw-read-request-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=0, AssignedPartitions=128}}
[WARN ] Coherence: 2012-01-26 10:56:20.121/333.315Oracle Coherence GE 3.7.1.2 <Warning> (thread=Recovery Thread, member=1): Attempting recovery of Guard{Daemon=DistributedCache:jdw-read-request-service}
[ERROR] Coherence: 2012-01-26 10:56:22.604/335.798Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=1): Detected hard timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:jdw-read-request-service} Service=PartitionedCache{Name=jdw-read-request-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=0, AssignedPartitions=128}}Now, for some reason (why?), Coherence determines that the service is unrecoverable and it must stop the cluster:
[WARN ] Coherence: 2012-01-26 10:56:22.605/335.799Oracle Coherence GE 3.7.1.2 <Warning> (thread=Termination Thread, member=1): Terminating Guard{Daemon=DistributedCache:jdw-read-request-service}
Coherence <Error>: Halting this cluster node due to unrecoverable service failure
[ERROR] Coherence: 2012-01-26 10:56:23.613/336.807Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=1): StopRunning ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1} due to unhandled exception:
[ERROR] Coherence: 2012-01-26 10:56:23.613/336.807Oracle Coherence GE 3.7.1.2 <Error> (thread=PacketListener1P, member=1): Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: UdpSocket.receive: unable to reopen socket; State=STATE_CLOSED
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:58)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Unknown Source)
[ERROR] Coherence: 2012-01-26 10:56:23.641/336.835Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=n/a):
java.lang.NullPointerException: null
at com.tangosol.coherence.component.net.Cluster$ClusterService$TcpRing.onAcceptException(Cluster.CDB:13) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.net.TcpRing.onAccept(TcpRing.CDB:25) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.net.TcpRing.onSelect(TcpRing.CDB:27) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.net.TcpRing.select(TcpRing.CDB:14) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onWait(ClusterService.CDB:6) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39) ~[coherence-3.7.1.2.jar:3.7.1.2]
at java.lang.Thread.run(Unknown Source) [na:1.6.0_30]
[WARN ] Coherence: 2012-01-26 10:56:24.976/338.170Oracle Coherence GE 3.7.1.2 <Warning> (thread=Invocation:jdw-invocation-service, member=n/a): failed to stop 95 worker threads; abandoning
Coherence <Error>: Halted the cluster:
Cluster is not running: State=5
Exception in thread "Cluster|SERVICE_STOPPED|Member(Id=1, Timestamp=2012-01-26 10:50:58.044, Address=192.168.1.67:9001, MachineId=52295, Location=site:,machine:DEN12956L,process:10012, Role=CoherenceServer)" java.nio.channels.ClosedSelector
Exception
at sun.nio.ch.SelectorImpl.keys(Unknown Source)
at com.tangosol.coherence.component.net.TcpRing.disconnectAll(TcpRing.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService$TcpRing.onLeft(ClusterService.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onStopRunning(ClusterService.CDB:7)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onException(ClusterService.CDB:28)
at com.tangosol.coherence.component.net.Cluster$ClusterService.onException(Cluster.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:85)
[ERROR] Coherence: 2012-01-26 10:56:25.616/338.810 Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=n/a): StopRunning ClusterService{Name=Cluster, State=(SERVICE_STOPPED, STATE_JOINED), Id=0, Version=3.7.1} due to unhandled exception:
at java.lang.Thread.run(Unknown Source)
[ERROR] Coherence: 2012-01-26 10:56:25.616/338.810Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=n/a):
java.nio.channels.ClosedSelectorException: null
at sun.nio.ch.SelectorImpl.keys(Unknown Source) ~[na:1.6.0_30]
at com.tangosol.coherence.component.net.TcpRing.close(TcpRing.CDB:11) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onExit(ClusterService.CDB:1) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:68) ~[coherence-3.7.1.2.jar:3.7.1.2]
at java.lang.Thread.run(Unknown Source) [na:1.6.0_30]Around the same time that member 1 decided to stop its cluster, member 2, which had been happily executing entry processors, begins taking responsibility for member 1's partitions. That's what I'd expect.
[WARN ] Coherence: 2012-01-26 10:56:23.643/336.784Oracle Coherence GE 3.7.1.2 <Warning> (thread=DistributedCache:sys-id-dist-service, member=2): Assigned 128 orphaned primary partitions
[WARN ] Coherence: 2012-01-26 10:56:23.646/336.787Oracle Coherence GE 3.7.1.2 <Warning> thread=DistributedCache:sourceMetadataReviewCache-service, member=2): Assigned 128 orphaned primary partitions
......Member 1 now restarts its cluster and re-joins as member 4, and starts re-starting its services:
[INFO ] Coherence: 2012-01-26 10:56:26.008/339.202Oracle Coherence GE 3.7.1.2 <Info> (thread=main, member=n/a): Restarting cluster
[INFO ] Coherence: 2012-01-26 10:56:26.327/339.521Oracle Coherence GE 3.7.1.2 <Info> (thread=Cluster, member=n/a): This Member(Id=4, Timestamp=2012-01-26 10:56:26.126, Address=192.168.1.67:9001, MachineId=52295, Location=site:,machine:DEN12956L,process:10012, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) joined cluster "NIR_GRID_DEV" with senior Member(Id=2, Timestamp=2012-01-26 10:51:03.593, Address=192.168.1.67:9003, MachineId=52295, Location=site:,machine:DEN12956L,process:10024, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
[INFO ] Coherence: 2012-01-26 10:56:26.364/339.558Oracle Coherence GE 3.7.1.2 <Info> (thread=main, member=n/a): Started cluster Name=NIR_GRID_DEV
Group{Address=224.3.7.0, Port=60000, TTL=0}
MasterMemberSet(
ThisMember=Member(Id=4, Timestamp=2012-01-26 10:56:26.126, Address=192.168.1.67:9001, MachineId=52295, Location=site:,machine:DEN12956L,process:10012, Role=CoherenceServer)
OldestMember=Member(Id=2, Timestamp=2012-01-26 10:51:03.593, Address=192.168.1.67:9003, MachineId=52295, Location=site:,machine:DEN12956L,process:10024, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=3
Member(Id=2, Timestamp=2012-01-26 10:51:03.593, Address=192.168.1.67:9003, MachineId=52295, Location=site:,machine:DEN12956L,process:10024, Role=CoherenceServer)
Member(Id=3, Timestamp=2012-01-26 10:55:05.522, Address=192.168.1.67:9005, MachineId=52295, Location=site:,machine:DEN12956L,process:13268, Role=IntellijRtExecutionAppMain)
Member(Id=4, Timestamp=2012-01-26 10:56:26.126, Address=192.168.1.67:9001, MachineId=52295, Location=site:,machine:DEN12956L,process:10012, Role=CoherenceServer)
MemberId|ServiceVersion|ServiceJoined|MemberState
2|3.7.1|2012-01-26 10:51:03.593|JOINED,
3|3.7.1|2012-01-26 10:55:05.522|JOINED,
4|3.7.1|2012-01-26 10:56:26.337|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
TcpRing{Connections=[2, 3]}
IpMonitor{AddressListSize=0}
[INFO ] Coherence: 2012-01-26 10:56:26.365/339.559Oracle Coherence GE 3.7.1.2 <Info> (thread=main, member=4): Restarting Service:Management
[INFO ] Coherence: 2012-01-26 10:56:26.417/339.611Oracle Coherence GE 3.7.1.2 <Info> (thread=main, member=4): Restarting Service:jdwSourceManagerCache-service
......While that is happening, member 2 now also encounters a hard timeout on the same service and begins going through the same process that member 1 just went through. I am not sure why member 2 encounters this timeout. Perhaps it is because it ran another entry processor that took too long? It is difficult to tell. Except this time we start having problems when member 2 tries to stop the cluster, and I'm not sure exactly why. Perhaps because member 4 is not yet in a good state?
[ERROR] Coherence: 2012-01-26 10:56:35.282/348.423Oracle Coherence GE 3.7.1.2 <Error> (thread=PacketListener1, member=2): Stopping cluster due to unhandled exception:
com.tangosol.net.messaging.ConnectionException: UdpSocket.receive: unable to reopen socket; State=STATE_CLOSED
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:58)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Unknown Source)
Coherence <Error>: Halted the cluster:
Cluster is not running: State=5
[ERROR] Coherence: 2012-01-26 10:56:42.147/355.288Oracle Coherence GE 3.7.1.2 <Error> (thread=main, member=n/a): Failed to restart services: com.tangosol.net.RequestTimeoutException: Timeout while waiting for cluster to stop.
[ERROR] Coherence: 2012-01-26 10:56:51.324/364.465Oracle Coherence GE 3.7.1.2 <Error> (thread=main, member=n/a): Failed to restart services: com.tangosol.net.RequestTimeoutException: Timeout while waiting for cluster to stop.
...... (error repeats)And the newly re-joined member 4, which had been restarting services and claiming partitions from the newly-dead member 2, appears to restart execution of the entry processor but then complains that its thread was interrupted: I am not sure why this happens or what it means.
[INFO ] Coherence: 2012-01-26 10:56:52.685/365.879Oracle Coherence GE 3.7.1.2 <Info> (thread=DistributedCache:jdw-read-request-service, member=4): Restarting NamedCache: coherence.common.sequencegenerators
[ERROR] Coherence: 2012-01-26 10:56:52.686/365.880Oracle Coherence GE 3.7.1.2 <Error> (thread=DistributedCache:jdw-read-request-service, member=4): This thread was interrupted while waiting for the results of a request:
Poll
PollId=2, active
InitTimeMillis=1327597012685
Service=DistributedCacheForSequenceGenerators (60)
RespondedMemberSet=[]
LeftMemberSet=[]
RemainingMemberSet=[4]
},Meanwhile, my client, who ran the invoke() with the EntryProcessor, has received a message stating "no storage enabled members exist," which although it may be true doesn't seem like it's really the cause of the problem.
Could I be having problems because all my cluster members are executing the same kind of long-running EntryProcessor and all of them are getting service guardian timeouts at around the same time? How do I deal with that? For example, what if a database is running very slowly one day and my entry processors all start to execute slowly? I don't want my whole grid having problems. Should I, for example, limit the number of concurrent EntryProcessors to < count of grid members by using a LimitFilter?
What does it mean that there was a timeout waiting for the cluster to stop?
And what about the client? If I get an exception after running a grid command, is it valid to wait a few seconds for things to stabilize and then try again? What's my best bet as far as that goes?
Thanks! And let me know if I should file a support request instead.rehevkor5 wrote:
I appreciate the responses so far, but unfortunately they give me no new information.
I am aware that I can try to avoid timeouts by using the PriorityTask API and heartbeats, but that still gives me no guarantee that I will not run into a deadlock situation and encounter a timeout. That is fine as long as my grid can deal with it. Currently, it does not behave well when that happens (all members eventually die). I challenge anyone reading this post to try running an entry processor on their grid that is designed to time out (not by using sleep() as that is interruptable via a soft timeout). All your members will probably encounter the timeout at the same time, and it probably will not end well. If, however, you can handle this situation or have some approach that mitigates it, I would love to hear from you!
I am also aware that I can disable the service guardians, or configure them to only log and take no action. However, this leaves me vulnerable to deadlocks.
Therefore, I would still appreciate any responses that address either of these two issues:
1. Configuration changes or other fixes that allow my members to recover successfully from a service timeout
2. Operational best practices that allow my grid to continue running even if an entry processor that is running on every member of the grid times out at the same time. For example, limiting the number of concurrent entry processors.One operational best practice is design your system so you can't get into a deadlock. If it can get into a deadlock once, it can get into it many times. If a system does that, it is not a well-behaving or well-designed system.
If your code can get into a deadlock across the cluster, that usually means that you are trying to do something which you should not have, like trying to operate on multiple resources in an arbitrary order while holding locks on them (e.g. violating the threading guidelines), which is the typical case of a distributed deadlock, and against which you can guard with ordering your multiple locking operations in a consistent order, but you should not expect Coherence to retry and possibly get into the same problem, as all you would achieve with it is that you converted the deadlock into a livelock.
Coherence is not supposed to guard against your errors. With the guardian it gives you a way to find out what your error is, but it should not attempt the impossible by trying to correct it. You should design your system so that it does not have this kinds of errors.
If you think there is a specific reproducible problem in how Coherence reacts to your code which can bring down the system, then create a test case and submit it in a SR. The kinds of exceptions you get make it possible that it may be necessary. If it is indeed Coherence's fault that it dies and not something you did, then I am fairly certain that it will be given quite high priority considering it is an error which can bring a cluster down.
Coherence server-side constructs have an API contract. Single foremost aspect of that is that the operations complete without an error (unless otherwise documented) within a small period of time. If it goes against this then it has to provide feedback or customization with PriorityTask/GuardSupport. A code which can deadlock goes against this contract. After all we are speaking about a grid, not a distributed process manager.
I agree that the grid should not die due to a deadlock, but you should not expect Coherence to fix a deadlock either. But dying due to an exception and having a distributed deadlock is two different and independent issues.
Best regards,
Robert
Edited by: robvarga on Feb 3, 2012 10:12 AM -
UdpSocket.receive: unable to reopen socket; State=STATE_CLOSED
Hi
We are getting this error on DefaultCacheServer instance while other members are loading information into Cache.
Thanks
sunder
2011-07-22 03:02:33.959/11191.128 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=16): StopRunning ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.6, OldestMemberId=1} due to unhandled exception:
2011-07-22 03:02:34.062/11191.231 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=16):
java.lang.NullPointerException
at sun.nio.ch.EPollSelectorImpl.implRegister(EPollSelectorImpl.java:149)
at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:115)
at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:180)
at com.tangosol.coherence.component.net.TcpRing.connect(TcpRing.CDB:20)
at com.tangosol.coherence.component.net.TcpRing.ensureTopology(TcpRing.CDB:39)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService$TcpRing.onMemberLeft(ClusterService.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onMemberLeft(ClusterService.CDB:27)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.removeMember(ClusterService.CDB:48)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.doMemberLeft(ClusterService.CDB:13)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onNotifyTcpDeparture(ClusterService.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService$TcpRing.onDeadBuddy(ClusterService.CDB:1)
at com.tangosol.coherence.component.net.TcpRing.heartbeat(TcpRing.CDB:36)
at com.tangosol.coherence.component.net.TcpRing.heartbeatBuddies(TcpRing.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onTimerRunning(ClusterService.CDB:61)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onNotify(ClusterService.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
2011-07-22 03:02:34.414/11191.583 Oracle Coherence GE 3.6.1.0 <Error> (thread=PacketListener1P, member=16): Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: UdpSocket.receive: unable to reopen socket; State=STATE_CLOSED
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:55)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)Appreciate response if someone encountered this problem before.
(Oracle Coherence - 3.7)
Thanks
sunder -
Spring with coherence cache deploy issue
hi, i configured spring with coherence cachestore, and i make s a sample example on that, load some value based key, if key not there in cache it automatically callbacks call to cachestore load(Object key) and ge5t value from db and put in cache........this application work in standardlone but if make web application application deploy time this error coming............
log4j:WARN No appenders could be found for logger (org.springframework.test.context.junit4.SpringJUnit4ClassRunner).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2013-10-10 16:05:09.528/1.279 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/D:/m2repo/org/tangosol/net/coherence/3.7/coherence-3.7.jar!/tangosol-coherence.xml"
2013-10-10 16:05:09.559/1.310 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/D:/m2repo/org/tangosol/net/coherence/3.7/coherence-3.7.jar!/tangosol-coherence-override-dev.xml"
2013-10-10 16:05:09.591/1.342 Oracle Coherence 3.7.1.1 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/D:/travererpibeproject/test/cnk-test-caching-web/src/main/webapp/WEB-INF/lib/cnk-common-caching-1.0.0-SNAPSHOT.jar!/tangosol-coherence-override.xml"
2013-10-10 16:05:09.591/1.342 Oracle Coherence 3.7.1.1 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.7.1.1 Build 28901
Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
2013-10-10 16:05:09.700/1.451 Oracle Coherence GE 3.7.1.1 <Info> (thread=main, member=n/a): Loaded cache configuration from "jar:file:/D:/travererpibeproject/test/cnk-test-caching-web/src/main/webapp/WEB-INF/lib/cnk-common-caching-1.0.0-SNAPSHOT.jar!/coherence-cache-config.xml"; this document does not refer to any schema definition and has not been validated.
cache service
2013-10-10 16:05:10.423/2.174 Oracle Coherence GE 3.7.1.1 <D4> (thread=main, member=n/a): TCMP bound to /10.21.12.29:8088 using SystemSocketProvider
2013-10-10 16:05:41.297/33.048 Oracle Coherence GE 3.7.1.1 <Info> (thread=Cluster, member=n/a): Created a new cluster "CoherenceCluster" with Member(Id=1, Timestamp=2013-10-10 16:05:10.564, Address=10.21.12.29:8088, MachineId=29338, Location=site:,machine:mumcnk-0562,process:2696, Role=EclipseJdtRemoteTestRunner, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) UID=0x0A150C1D00000141A1EF3564729A1F98
2013-10-10 16:05:41.297/33.048 Oracle Coherence GE 3.7.1.1 <Info> (thread=main, member=n/a): Started cluster Name=CoherenceCluster
Group{Address=231.1.1.1, Port=7777, TTL=3}
MasterMemberSet(
ThisMember=Member(Id=1, Timestamp=2013-10-10 16:05:10.564, Address=10.21.12.29:8088, MachineId=29338, Location=site:,machine:mumcnk-0562,process:2696, Role=EclipseJdtRemoteTestRunner)
OldestMember=Member(Id=1, Timestamp=2013-10-10 16:05:10.564, Address=10.21.12.29:8088, MachineId=29338, Location=site:,machine:mumcnk-0562,process:2696, Role=EclipseJdtRemoteTestRunner)
ActualMemberSet=MemberSet(Size=1
Member(Id=1, Timestamp=2013-10-10 16:05:10.564, Address=10.21.12.29:8088, MachineId=29338, Location=site:,machine:mumcnk-0562,process:2696, Role=EclipseJdtRemoteTestRunner)
MemberId|ServiceVersion|ServiceJoined|MemberState
1|3.7.1|2013-10-10 16:05:41.297|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}
2013-10-10 16:05:41.329/33.080 Oracle Coherence GE 3.7.1.1 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
2013-10-10 16:05:41.548/33.299 Oracle Coherence GE 3.7.1.1 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 1
2013-10-10 16:10:16.093/307.844 Oracle Coherence GE 3.7.1.1 <Error> (thread=Cluster, member=1): Detected soft timeout) of {WrapperGuardable Guard{Daemon=DistributedCache} Service=PartitionedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}}
2013-10-10 16:10:16.093/307.844 Oracle Coherence GE 3.7.1.1 <Error> (thread=Recovery Thread, member=1): Full Thread Dump
Thread[Logger@9254847 3.7.1.1,3,main]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[Attach Listener,5,system]
Thread[Signal Dispatcher,9,system]
Thread[ReaderThread,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:150)
java.net.SocketInputStream.read(SocketInputStream.java:121)
sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
java.io.InputStreamReader.read(InputStreamReader.java:184)
java.io.BufferedReader.fill(BufferedReader.java:154)
java.io.BufferedReader.readLine(BufferedReader.java:317)
java.io.BufferedReader.readLine(BufferedReader.java:382)
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:140)
Thread[Invocation:Management:EventDispatcher,10,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[Invocation:Management,10,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onWait(Service.CDB:4)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:3)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[PacketReceiver,10,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onWait(PacketReceiver.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[Cluster|Member(Id=1, Timestamp=2013-10-10 16:05:10.564, Address=10.21.12.29:8088, MachineId=29338, Location=site:,machine:mumcnk-0562,process:2696, Role=EclipseJdtRemoteTestRunner),5,Cluster]
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
com.tangosol.coherence.component.net.TcpRing.select(TcpRing.CDB:11)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onWait(ClusterService.CDB:6)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[Reference Handler,10,system]
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
Thread[DistributedCache,10,Cluster]
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:184)
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:169)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:237)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
com.cnk.travelerp.common.caching.service.SpringAwareCacheFactory.instantiateAny(SpringAwareCacheFactory.java:161)
com.tangosol.net.DefaultConfigurableCacheFactory.instantiateCacheStore(DefaultConfigurableCacheFactory.java:3324)
com.tangosol.net.DefaultConfigurableCacheFactory.instantiateReadWriteBackingMap(DefaultConfigurableCacheFactory.java:1753)
com.tangosol.net.DefaultConfigurableCacheFactory.configureBackingMap(DefaultConfigurableCacheFactory.java:1500)
com.tangosol.net.DefaultConfigurableCacheFactory$Manager.instantiateBackingMap(DefaultConfigurableCacheFactory.java:4111)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.instantiateBackingMap(PartitionedCache.CDB:23)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.setCacheName(PartitionedCache.CDB:25)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ServiceConfig$ConfigListener.entryInserted(PartitionedCache.CDB:17)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:266)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:567)
com.tangosol.util.ObservableHashMap.dispatchEvent(ObservableHashMap.java:229)
com.tangosol.util.ObservableHashMap$Entry.onAdd(ObservableHashMap.java:270)
com.tangosol.util.SafeHashMap.put(SafeHashMap.java:244)
com.tangosol.coherence.component.util.ServiceConfig$Map.put(ServiceConfig.CDB:43)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$StorageIdRequest.onReceived(PartitionedCache.CDB:45)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:34)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[Recovery Thread,5,Cluster]
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Thread.java:1618)
com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:810)
com.tangosol.internal.net.cluster.DefaultServiceFailurePolicy.onGuardableRecovery(DefaultServiceFailurePolicy.java:44)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$WrapperGuardable.recover(Grid.CDB:1)
com.tangosol.net.GuardSupport$Context$1.run(GuardSupport.java:653)
java.lang.Thread.run(Thread.java:722)
Thread[PacketListenerN,10,Cluster]
java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)
java.net.TwoStacksPlainDatagramSocketImpl.receive(TwoStacksPlainDatagramSocketImpl.java:90)
java.net.DatagramSocket.receive(DatagramSocket.java:786)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[PacketPublisher,10,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onWait(PacketPublisher.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[PacketListener1P,10,Cluster]
java.net.DualStackPlainDatagramSocketImpl.socketReceiveOrPeekData(Native Method)
java.net.DualStackPlainDatagramSocketImpl.receive0(DualStackPlainDatagramSocketImpl.java:105)
java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
java.net.DatagramSocket.receive(DatagramSocket.java:786)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[main,5,main]
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:26)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.ensureCache(PartitionedCache.CDB:29)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.ensureCache(PartitionedCache.CDB:36)
com.tangosol.coherence.component.util.safeService.SafeCacheService.ensureCache$Router(SafeCacheService.CDB:1)
com.tangosol.coherence.component.util.safeService.SafeCacheService.ensureCache(SafeCacheService.CDB:33)
com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:920)
com.tangosol.net.DefaultConfigurableCacheFactory.configureCache(DefaultConfigurableCacheFactory.java:1296)
com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:297)
com.tangosol.net.CacheFactory.getCache(CacheFactory.java:204)
com.tangosol.net.CacheFactory.getCache(CacheFactory.java:181)
com.cnk.travelerp.common.caching.service.CacheService.<init>(CacheService.java:36)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:525)
org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:76)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:990)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:943)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:876)
org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:818)
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:735)
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:478)
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:284)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1106)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:609)
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:469)
org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:103)
org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:1)
org.springframework.test.context.support.DelegatingSmartContextLoader.loadContext(DelegatingSmartContextLoader.java:228)
org.springframework.test.context.TestContext.loadApplicationContext(TestContext.java:124)
org.springframework.test.context.TestContext.getApplicationContext(TestContext.java:148)
org.springframework.test.context.support.DependencyInjectionTestExecutionListener.injectDependencies(DependencyInjectionTestExecutionListener.java:109)
org.springframework.test.context.support.DependencyInjectionTestExecutionListener.prepareTestInstance(DependencyInjectionTestExecutionListener.java:75)
org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:321)
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:211)
org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:288)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:290)
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:231)
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
org.junit.runners.ParentRunner.run(ParentRunner.java:236)
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:174)
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Thread[PacketListener1,10,Cluster]
java.net.DualStackPlainDatagramSocketImpl.socketReceiveOrPeekData(Native Method)
java.net.DualStackPlainDatagramSocketImpl.receive0(DualStackPlainDatagramSocketImpl.java:105)
java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
java.net.DatagramSocket.receive(DatagramSocket.java:786)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[IpMonitor,6,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.IpMonitor.onWait(IpMonitor.CDB:4)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[PacketSpeaker,10,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketSpeaker.onNotify(PacketSpeaker.CDB:21)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[Finalizer,8,system]
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
2013-10-10 16:10:16.093/307.844 Oracle Coherence GE 3.7.1.1 <Warning> (thread=Recovery Thread, member=1): Attempting recovery of Guard{Daemon=DistributedCache}
2013-10-10 16:10:46.607/338.358 Oracle Coherence GE 3.7.1.1 <Error> (thread=Cluster, member=1): Detected hard timeout) of {WrapperGuardable Guard{Daemon=DistributedCache} Service=PartitionedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}}
2013-10-10 16:10:46.607/338.358 Oracle Coherence GE 3.7.1.1 <Error> (thread=Termination Thread, member=1): Full Thread Dump
Thread[Logger@9254847 3.7.1.1,3,main]
java.io.FileOutputStream.writeBytes(Native Method)
java.io.FileOutputStream.write(FileOutputStream.java:318)
java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
java.io.PrintStream.write(PrintStream.java:480)
sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
java.io.PrintStream.write(PrintStream.java:527)
java.io.PrintStream.print(PrintStream.java:669)
java.io.PrintStream.println(PrintStream.java:806)
com.tangosol.coherence.component.util.logOutput.Standard.log(Standard.CDB:9)
com.tangosol.coherence.component.util.logOutput.Standard.log(Standard.CDB:1)
com.tangosol.coherence.component.util.LogOutput.log(LogOutput.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.Logger.onNotify(Logger.CDB:99)
com.tangosol.coherence.component.application.console.Coherence$Logger.onNotify(Coherence.CDB:4)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[Attach Listener,5,system]
Thread[Signal Dispatcher,9,system]
Thread[ReaderThread,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:150)
java.net.SocketInputStream.read(SocketInputStream.java:121)
sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
java.io.InputStreamReader.read(InputStreamReader.java:184)
java.io.BufferedReader.fill(BufferedReader.java:154)
java.io.BufferedReader.readLine(BufferedReader.java:317)
java.io.BufferedReader.readLine(BufferedReader.java:382)
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:140)
Thread[Termination Thread,5,Cluster]
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Thread.java:1618)
com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:810)
com.tangosol.internal.net.cluster.DefaultServiceFailurePolicy.onGuardableTerminate(DefaultServiceFailurePolicy.java:80)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$WrapperGuardable.terminate(Grid.CDB:1)
com.tangosol.net.GuardSupport$Context$2.run(GuardSupport.java:677)
java.lang.Thread.run(Thread.java:722)
Thread[Invocation:Management:EventDispatcher,10,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[Invocation:Management,10,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onWait(Service.CDB:4)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:3)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[PacketReceiver,10,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onWait(PacketReceiver.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[Cluster|Member(Id=1, Timestamp=2013-10-10 16:05:10.564, Address=10.21.12.29:8088, MachineId=29338, Location=site:,machine:mumcnk-0562,process:2696, Role=EclipseJdtRemoteTestRunner),5,Cluster]
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
com.tangosol.coherence.component.net.TcpRing.select(TcpRing.CDB:11)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onWait(ClusterService.CDB:6)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[Reference Handler,10,system]
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
Thread[DistributedCache,10,Cluster]
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:184)
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:169)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:237)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
com.cnk.travelerp.common.caching.service.SpringAwareCacheFactory.instantiateAny(SpringAwareCacheFactory.java:161)
com.tangosol.net.DefaultConfigurableCacheFactory.instantiateCacheStore(DefaultConfigurableCacheFactory.java:3324)
com.tangosol.net.DefaultConfigurableCacheFactory.instantiateReadWriteBackingMap(DefaultConfigurableCacheFactory.java:1753)
com.tangosol.net.DefaultConfigurableCacheFactory.configureBackingMap(DefaultConfigurableCacheFactory.java:1500)
com.tangosol.net.DefaultConfigurableCacheFactory$Manager.instantiateBackingMap(DefaultConfigurableCacheFactory.java:4111)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.instantiateBackingMap(PartitionedCache.CDB:23)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.setCacheName(PartitionedCache.CDB:25)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ServiceConfig$ConfigListener.entryInserted(PartitionedCache.CDB:17)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:266)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:567)
com.tangosol.util.ObservableHashMap.dispatchEvent(ObservableHashMap.java:229)
com.tangosol.util.ObservableHashMap$Entry.onAdd(ObservableHashMap.java:270)
com.tangosol.util.SafeHashMap.put(SafeHashMap.java:244)
com.tangosol.coherence.component.util.ServiceConfig$Map.put(ServiceConfig.CDB:43)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$StorageIdRequest.onReceived(PartitionedCache.CDB:45)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:34)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[PacketListenerN,10,Cluster]
java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)
java.net.TwoStacksPlainDatagramSocketImpl.receive(TwoStacksPlainDatagramSocketImpl.java:90)
java.net.DatagramSocket.receive(DatagramSocket.java:786)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[PacketPublisher,10,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onWait(PacketPublisher.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[PacketListener1P,10,Cluster]
java.net.DualStackPlainDatagramSocketImpl.socketReceiveOrPeekData(Native Method)
java.net.DualStackPlainDatagramSocketImpl.receive0(DualStackPlainDatagramSocketImpl.java:105)
java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
java.net.DatagramSocket.receive(DatagramSocket.java:786)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[main,5,main]
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:26)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:11)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.ensureCache(PartitionedCache.CDB:29)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.ensureCache(PartitionedCache.CDB:36)
com.tangosol.coherence.component.util.safeService.SafeCacheService.ensureCache$Router(SafeCacheService.CDB:1)
com.tangosol.coherence.component.util.safeService.SafeCacheService.ensureCache(SafeCacheService.CDB:33)
com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:920)
com.tangosol.net.DefaultConfigurableCacheFactory.configureCache(DefaultConfigurableCacheFactory.java:1296)
com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:297)
com.tangosol.net.CacheFactory.getCache(CacheFactory.java:204)
com.tangosol.net.CacheFactory.getCache(CacheFactory.java:181)
com.cnk.travelerp.common.caching.service.CacheService.<init>(CacheService.java:36)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:525)
org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:76)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:990)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:943)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:876)
org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:818)
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:735)
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:478)
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:284)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1106)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:609)
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:469)
org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:103)
org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:1)
org.springframework.test.context.support.DelegatingSmartContextLoader.loadContext(DelegatingSmartContextLoader.java:228)
org.springframework.test.context.TestContext.loadApplicationContext(TestContext.java:124)
org.springframework.test.context.TestContext.getApplicationContext(TestContext.java:148)
org.springframework.test.context.support.DependencyInjectionTestExecutionListener.injectDependencies(DependencyInjectionTestExecutionListener.java:109)
org.springframework.test.context.support.DependencyInjectionTestExecutionListener.prepareTestInstance(DependencyInjectionTestExecutionListener.java:75)
org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:321)
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:211)
org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:288)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:290)
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:231)
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
org.junit.runners.ParentRunner.run(ParentRunner.java:236)
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:174)
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Thread[PacketListener1,10,Cluster]
java.net.DualStackPlainDatagramSocketImpl.socketReceiveOrPeekData(Native Method)
java.net.DualStackPlainDatagramSocketImpl.receive0(DualStackPlainDatagramSocketImpl.java:105)
java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
java.net.DatagramSocket.receive(DatagramSocket.java:786)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[IpMonitor,6,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.IpMonitor.onWait(IpMonitor.CDB:4)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:722)
Thread[PacketSpeaker,10,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketSpeaker.onNotify(PacketSpeaker.CDB:21)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:722)
Thread[Finalizer,8,system]
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
2013-10-10 16:10:46.607/338.358 Oracle Coherence GE 3.7.1.1 <Warning> (thread=Termination Thread, member=1): Terminating Guard{Daemon=DistributedCache}
Coherence <Error>: Halting this cluster node due to unrecoverable service failure
2013-10-10 16:10:47.622/339.373 Oracle Coherence GE 3.7.1.1 <Error> (thread=PacketListener1P, member=1): Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: UdpSocket.receive: unable to reopen socket; State=STATE_CLOSED
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:58)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:722)
2013-10-10 16:10:47.622/339.373 Oracle Coherence GE 3.7.1.1 <D5> (thread=Cluster, member=n/a): Unexpected exception java.nio.channels.ClosedChannelException while accepting incoming TcpRing connection; refreshing listener socket
2013-10-10 16:10:47.622/339.373 Oracle Coherence GE 3.7.1.1 <D5> (thread=Invocation:Management, member=n/a): Service Management left the cluster
2013-10-10 16:10:47.622/339.373 Oracle Coherence GE 3.7.1.1 <Error> (thread=Cluster, member=n/a): StopRunning ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1} due to unhandled exception:
2013-10-10 16:10:47.622/339.373 Oracle Coherence GE 3.7.1.1 <Error> (thread=Cluster, member=n/a):
java.lang.NullPointerException
at com.tangosol.coherence.component.net.Cluster$ClusterService$TcpRing.onAcceptException(Cluster.CDB:13)
at com.tangosol.coherence.component.net.TcpRing.onAccept(TcpRing.CDB:25)
at com.tangosol.coherence.component.net.TcpRing.onSelect(TcpRing.CDB:27)
at com.tangosol.coherence.component.net.TcpRing.select(TcpRing.CDB:14)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onWait(ClusterService.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
at java.lang.Thread.run(Thread.java:722)
2013-10-10 16:10:49.636/341.387 Oracle Coherence GE 3.7.1.1 <D5> (thread=Cluster, member=n/a): Service Cluster left the cluster
2013-10-10 16:10:49.636/341.387 Oracle Coherence GE 3.7.1.1 <Error> (thread=Cluster, member=n/a): StopRunning ClusterService{Name=Cluster, State=(SERVICE_STOPPED, STATE_JOINED), Id=0, Version=3.7.1} due to unhandled exception:
2013-10-10 16:10:49.636/341.387 Oracle Coherence GE 3.7.1.1 <Error> (thread=Cluster, member=n/a):
java.nio.channels.ClosedSelectorException
at sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:69)
at com.tangosol.coherence.component.net.TcpRing.close(TcpRing.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onExit(ClusterService.CDB:1)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:68)
at java.lang.Thread.run(Thread.java:722)
Exception in thread "Cluster|SERVICE_STOPPED|Member(Id=1, Timestamp=2013-10-10 16:05:10.564, Address=10.21.12.29:8088, MachineId=29338, Location=site:,machine:mumcnk-0562,process:2696, Role=EclipseJdtRemoteTestRunner)" java.nio.channels.ClosedSelectorException
at sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:69)
at com.tangosol.coherence.component.net.TcpRing.disconnectAll(TcpRing.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService$TcpRing.onLeft(ClusterService.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onStopRunning(ClusterService.CDB:7)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onException(ClusterService.CDB:28)
at com.tangosol.coherence.component.net.Cluster$ClusterService.onException(Cluster.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:85)
at java.lang.Thread.run(Thread.java:722)Hi,
Did you check the documentation about how to integrate Coherence with Spring applications? Check it out: Integrating Spring with Coherence
It seems that you are experiencing a communication problem between your web container and the Coherence cluster. Check if you are able to access simple caches and execute sample put() and get()s. Maybe the problem is not about Spring itself, it could be a communication problem.
Cheers,
Ricardo Ferreira -
Could you please let me know what is wrong with my code.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>cacheEquityMarket</cache-name>
<scheme-name>distributed-cache1</scheme-name>
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
<param-name>local-storage</param-name>
<param-value system-property="cacheEquityMarket.localstorage">true</param-value>
</init-param>
</init-params>
</cache-mapping>
<cache-mapping>
<cache-name>cacheEquityCompany</cache-name>
<scheme-name>distributed-cache2</scheme-name>
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
<param-name>local-storage</param-name>
<param-value system-property="cacheEquityCompany.localstorage">true</param-value>
</init-param>
</init-params>
</cache-mapping>
<cache-mapping>
<cache-name>cacheEquityHistorical</cache-name>
<scheme-name>distributed-cache3</scheme-name>
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
<param-name>local-storage</param-name>
<param-value system-property="cacheEquityHistorical.localstorage">true</param-value>
</init-param>
</init-params>
</cache-mapping>
<cache-mapping>
<cache-name>cacheEquityIPO</cache-name>
<scheme-name>distributed-cache4</scheme-name>
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
<param-name>local-storage</param-name>
<param-value system-property="cacheEquityIPO.localstorage">true</param-value>
</init-param>
</init-params>
</cache-mapping>
<cache-mapping>
<cache-name>cacheDerivativeMaster</cache-name>
<scheme-name>distributed-cache5</scheme-name>
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
<param-name>local-storage</param-name>
<param-value system-property="cacheDerivativeMaster.localstorage">true</param-value>
</init-param>
</init-params>
</cache-mapping>
<cache-mapping>
<cache-name>cacheDerivativeMarket</cache-name>
<scheme-name>distributed-cache6</scheme-name>
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
<param-name>local-storage</param-name>
<param-value system-property="cacheDerivativeMarket.localstorage">true</param-value>
</init-param>
</init-params>
</cache-mapping>
<cache-mapping>
<cache-name>cacheDerivativeHistorical</cache-name>
<scheme-name>distributed-cache7</scheme-name>
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
<param-name>local-storage</param-name>
<param-value system-property="cacheDerivativeHistorical.localstorage">true</param-value>
</init-param>
</init-params>
</cache-mapping>
<cache-mapping>
<cache-name>cacheMFMaster</cache-name>
<scheme-name>distributed-cache8</scheme-name>
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
<param-name>local-storage</param-name>
<param-value system-property="cacheMFMaster.localstorage">true</param-value>
</init-param>
</init-params>
</cache-mapping>
<cache-mapping>
<cache-name>cacheMFData</cache-name>
<scheme-name>distributed-cache9</scheme-name>
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
<param-name>local-storage</param-name>
<param-value system-property="cacheMFData.localstorage">true</param-value>
</init-param>
</init-params>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>base-entity-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
<scheme-name>distributed-cache1</scheme-name>
<service-name>DistributedEqMktService</service-name>
<!--
serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params> <init-param> <param-type>java.lang.String</param-type>
<param-value>base-entity-pof-config.xml</param-value> </init-param>
</init-params> </serializer
-->
<internal-cache-scheme>
<local-scheme>
<high-units>1000</high-units>
<expiry-delay>1m</expiry-delay>
</local-scheme>
</internal-cache-scheme>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>com.tangosol.util.ObservableHashMap</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com.fs.framework.cache.cachestores.EquityMarketCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<refresh-ahead-factor>0.5</refresh-ahead-factor>
<read-only>false</read-only>
<expiry-delay>10s</expiry-delay>
<flush-delay>2s</flush-delay>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
<distributed-scheme>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>base-entity-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
<scheme-name>distributed-cache2</scheme-name>
<service-name>DistributedEqCoyService</service-name>
<internal-cache-scheme>
<local-scheme>
<high-units>1000</high-units>
<expiry-delay>1m</expiry-delay>
</local-scheme>
</internal-cache-scheme>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>com.tangosol.util.ObservableHashMap</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com..fs.framework.cache.cachestores.EquityCompanyCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<refresh-ahead-factor>0.5</refresh-ahead-factor>
<read-only>false</read-only>
<expiry-delay>10s</expiry-delay>
<flush-delay>2s</flush-delay>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
<distributed-scheme>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>base-entity-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
<scheme-name>distributed-cache3</scheme-name>
<service-name>DistributedEqHistService</service-name>
<internal-cache-scheme>
<local-scheme>
<high-units>1000</high-units>
<expiry-delay>1m</expiry-delay>
</local-scheme>
</internal-cache-scheme>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>com.tangosol.util.ObservableHashMap</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com..fs.framework.cache.cachestores.EquityHistoricalCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<refresh-ahead-factor>0.5</refresh-ahead-factor>
<read-only>false</read-only>
<expiry-delay>10s</expiry-delay>
<flush-delay>2s</flush-delay>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
<distributed-scheme>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>base-entity-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
<scheme-name>distributed-cache4</scheme-name>
<service-name>DistributedEqIPOService</service-name>
<internal-cache-scheme>
<local-scheme>
<high-units>1000</high-units>
<expiry-delay>1m</expiry-delay>
</local-scheme>
</internal-cache-scheme>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>com.tangosol.util.ObservableHashMap</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com..fs.framework.cache.cachestores.EquityIPOCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<refresh-ahead-factor>0.5</refresh-ahead-factor>
<read-only>false</read-only>
<expiry-delay>10s</expiry-delay>
<flush-delay>2s</flush-delay>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
<distributed-scheme>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>base-entity-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
<scheme-name>distributed-cache5</scheme-name>
<service-name>DistributedDerivativeMstrService</service-name>
<internal-cache-scheme>
<local-scheme>
<high-units>1000</high-units>
<expiry-delay>1m</expiry-delay>
</local-scheme>
</internal-cache-scheme>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>com.tangosol.util.ObservableHashMap</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com..fs.framework.cache.cachestores.DerivativeMasterCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<refresh-ahead-factor>0.5</refresh-ahead-factor>
<read-only>false</read-only>
<expiry-delay>10s</expiry-delay>
<flush-delay>2s</flush-delay>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
<distributed-scheme>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>base-entity-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
<scheme-name>distributed-cache6</scheme-name>
<service-name>DistributedDerivativeMktService</service-name>
<internal-cache-scheme>
<local-scheme>
<high-units>1000</high-units>
<expiry-delay>1m</expiry-delay>
</local-scheme>
</internal-cache-scheme>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>com.tangosol.util.ObservableHashMap</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com.fs.framework.cache.cachestores.DerivativeMarketCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<refresh-ahead-factor>0.5</refresh-ahead-factor>
<read-only>false</read-only>
<expiry-delay>10s</expiry-delay>
<flush-delay>2s</flush-delay>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
<distributed-scheme>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>base-entity-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
<scheme-name>distributed-cache7</scheme-name>
<service-name>DistributedDerivativeHistService</service-name>
<internal-cache-scheme>
<local-scheme>
<high-units>1000</high-units>
<expiry-delay>1m</expiry-delay>
</local-scheme>
</internal-cache-scheme>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>com.tangosol.util.ObservableHashMap</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com.fs.framework.cache.cachestores.DerivativeHistoricalCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<refresh-ahead-factor>0.5</refresh-ahead-factor>
<read-only>false</read-only>
<expiry-delay>10s</expiry-delay>
<flush-delay>2s</flush-delay>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
<distributed-scheme>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>base-entity-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
<scheme-name>distributed-cache8</scheme-name>
<service-name>DistributedMFMstrService</service-name>
<internal-cache-scheme>
<local-scheme>
<high-units>1000</high-units>
<expiry-delay>1m</expiry-delay>
</local-scheme>
</internal-cache-scheme>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>com.tangosol.util.ObservableHashMap</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com.fs.framework.cache.cachestores.MFMasterCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<refresh-ahead-factor>0.5</refresh-ahead-factor>
<read-only>false</read-only>
<expiry-delay>10s</expiry-delay>
<flush-delay>2s</flush-delay>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
<distributed-scheme>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>base-entity-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
<scheme-name>distributed-cache9</scheme-name>
<service-name>DistributedMFDataService</service-name>
<internal-cache-scheme>
<local-scheme>
<high-units>1000</high-units>
<expiry-delay>1m</expiry-delay>
</local-scheme>
</internal-cache-scheme>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>com.tangosol.util.ObservableHashMap</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com.fs.framework.cache.cachestores.MFDataCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<refresh-ahead-factor>0.5</refresh-ahead-factor>
<read-only>false</read-only>
<expiry-delay>10s</expiry-delay>
<flush-delay>2s</flush-delay>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
</caching-schemes>
</cache-config>
<?xml version="1.0"?>
<!DOCTYPE pof-config SYSTEM "pof-config.dtd">
<pof-config>
<user-type-list>
<!-- coherence POF user types -->
<include>coherence-pof-config.xml</include>
<!-- com.tangosol.examples package -->
<user-type>
<type-id>1001</type-id>
<class-name>com.fs.framework.cache.model.BaseCacheEntity</class-name>
</user-type>
</user-type-list>
<allow-interfaces>true</allow-interfaces>
<allow-subclasses>true</allow-subclasses>
</pof-config>
package com.fs.framework.cache.model;
import java.io.IOException;
import java.io.Serializable;
import org.apache.log4j.Logger;
import com.tangosol.io.pof.PofReader;
import com.tangosol.io.pof.PortableObject;
import com.tangosol.io.pof.PofWriter;
import com.tangosol.util.Base;
public class BaseCacheEntity implements PortableObject{
public BaseCacheEntity() {
// TODO Auto-generated constructor stub
public BaseCacheEntity(String dataType, String data) {
super();
this.dataType = dataType;
this.data = data;
private static Logger logger = Logger.getLogger(BaseCacheEntity.class);
private String dataType;
private String data;
public String getDataType() {
return dataType;
public void setDataType(String dataType) {
this.dataType = dataType;
public String getData() {
return data;
public void setData(String data) {
this.data = data;
@Override
public void readExternal(PofReader pofReader) throws IOException {
// TODO Auto-generated method stub
this.setData((String)pofReader.readObject(0));
this.setDataType((String)pofReader.readObject(1));
@Override
public void writeExternal(PofWriter pofWriter) throws IOException {
// TODO Auto-generated method stub
pofWriter.writeObject(0, this.getData());
pofWriter.writeObject(1, this.getDataType());
2010-06-24 21:13:33.861/211.255 Oracle Coherence GE 3.5.3/465 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2010-06-24 21:13:34.064/211.458 Oracle Coherence GE 3.5.3/465 <Info> (thread=Cluster, member=n/a): This Member(Id=3, Timestamp=2010-06-24 21:13:33.587, Address=172.17.89.51:8088, MachineId=58419, Location=site:India.com,machine:01HW136561,process:4124, Role=WeblogicServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=1) joined cluster "cluster:0xDDEB" with senior Member(Id=2, Timestamp=2010-06-22 19:22:43.104, Address=172.17.88.171:8089, MachineId=58795, Location=site:India.com,machine:01HW133294,process:1304, Role=CoherenceConsole, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=1)
2010-06-24 21:13:34.080/211.474 Oracle Coherence GE 3.5.3/465 <Error> (thread=Cluster, member=n/a): An exception (java.lang.IllegalStateException) occurred reading Message MemberConfigResponse Type=-2 for Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=2}
2010-06-24 21:13:34.080/211.474 Oracle Coherence GE 3.5.3/465 <Error> (thread=Cluster, member=n/a): StopRunning ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=2} due to unhandled exception:
2010-06-24 21:13:34.080/211.474 Oracle Coherence GE 3.5.3/465 <Error> (thread=Cluster, member=n/a):
java.lang.IllegalStateException: Missing POF configuration (Config=C:\com\fs-base-framework\src\resources\base-entity-pof-config.xml)
at com.tangosol.io.pof.ConfigurablePofContext.report(ConfigurablePofContext.java:1311)
at com.tangosol.io.pof.ConfigurablePofContext.createPofConfig(ConfigurablePofContext.java:803)
at com.tangosol.io.pof.ConfigurablePofContext.initialize(ConfigurablePofContext.java:762)
at com.tangosol.io.pof.ConfigurablePofContext.setContextClassLoader(ConfigurablePofContext.java:325)
at com.tangosol.util.ExternalizableHelper.ensureSerializer(ExternalizableHelper.java:286)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.ensureSerializer(Service.CDB:25)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.ensureSerializer(Service.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.readObject(Service.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$ServiceConfigMap.readObject(Grid.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$MemberConfigResponse.read(Grid.CDB:13)
Edited by: tally_15 on Jun 24, 2010 8:54 AMI believe there are some other concerns as well
<cache-mapping>
<cache-name>cacheEquityMarket</cache-name>
<scheme-name>distributed-cache1</scheme-name>
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
<param-name>local-storage</param-name>
<param-value system-property="cacheEquityMarket.localstorage">true</param-value>
</init-param>
</init-params>
</cache-mapping>
The cavhe-config.dtd shows the cache-mapping as
<!ELEMENT cache-mapping
(cache-name, scheme-name, init-params?)>
Where init-params is defined as
!ELEMENT init-param
((param-name | param-type), param-value)>
Which states either have a param-name or a param-type with a param-value
Which would change the above xml to:
<init-params>
<init-param>
<param-name>java.lang.String</param-name>
<param-value>{cache-name}</param-value>
</init-param>
<init-param>
<param-name>local-storage</param-name>
<param-value system-property="cacheEquityMarket.localstorage">true</param-value>
</init-param>
</init-params>
I believe the java.lang.String is meant to be a <param-type> instead of <param-name>
which would change the xml to:
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
<init-param>
<param-name>local-storage</param-name>
<param-value system-property="cacheEquityMarket.localstorage">true</param-value>
</init-param>
</init-params>
These are different examples from the 3.5 user guide
<class-name>com.mycompany.cache.CustomCacheLoader</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>EmployeeTable</param-value>
</init-param>
<init-param>
<param-type>int</param-type>
<param-value>2000</param-value>
</init-param>
</init-params>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
Edited by: drowland on Jun 24, 2010 12:06 PM
Edited by: drowland on Jun 24, 2010 12:24 PM -
Unable to load Custom POF Configuration
I successfully downloaded Coherence 3.7 and able to run the from .Net client with default Coherence.POF.Config file. As soon as I'm changing the POF file configuration to custom type, I'm unable to run the Coherence server. And finding the following error. Any Help is appreciated.
2014-07-25 18:31:07.130/4.930 Oracle Coherence GE 3.7.1.0 <Info> (thread=Cluster
, member=1): Loaded POF configuration from "jar:file:/C:/OracleCoherence/coheren
ce/lib/coherence.jar!/pof-config.xml"
2014-07-25 18:31:07.147/4.947 Oracle Coherence GE 3.7.1.0 <Info> (thread=Cluster
, member=1): Loaded included POF configuration from "jar:file:/C:/OracleCoherenc
e/coherence/lib/coherence.jar!/coherence-pof-config.xml"
2014-07-25 18:31:07.194/4.994 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocatio
n:Management, member=1): Service Management joined the cluster with senior servi
ce member 1
2014-07-25 18:31:07.401/5.201 Oracle Coherence GE 3.7.1.0 <D5> (thread=Distribut
edCache, member=1): Service DistributedCache joined the cluster with senior serv
ice member 1
Exception in thread "main" (Wrapped: error creating class "com.tangosol.io.pof.C
onfigurablePofContext") (Wrapped: Failed to load POF configuration: Mypof-config
.xml) java.io.IOException: The POF configuration is missing: "Mypof-config.xml",
loader=sun.misc.Launcher$AppClassLoader@1b90b39
at com.tangosol.io.ConfigurableSerializerFactory.createSerializer(Config
urableSerializerFactory.java:46)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.i
nstantiateSerializer(Service.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.e
nsureSerializer(Service.CDB:32)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.e
nsureSerializer(Service.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ProxyService.configure(ProxyService.CDB:101)
at com.tangosol.coherence.component.util.SafeService.startService(SafeSe
rvice.CDB:17)
at com.tangosol.coherence.component.util.SafeService.ensureRunningServic
e(SafeService.CDB:27)
at com.tangosol.coherence.component.util.SafeService.start(SafeService.C
DB:14)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInterna
l(DefaultConfigurableCacheFactory.java:1105)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(Defaul
tConfigurableCacheFactory.java:937)
at com.tangosol.net.DefaultCacheServer.startServices(DefaultCacheServer.
java:81)
at com.tangosol.net.DefaultCacheServer.intialStartServices(DefaultCacheS
erver.java:250)
at com.tangosol.net.DefaultCacheServer.startAndMonitor(DefaultCacheServe
r.java:55)
at com.tangosol.net.DefaultCacheServer.main(DefaultCacheServer.java:197)
Caused by: (Wrapped: Failed to load POF configuration: Mypof-config.xml) java.io
.IOException: The POF configuration is missing: "Mypof-config.xml", loader=sun.m
isc.Launcher$AppClassLoader@1b90b39
at com.tangosol.util.Base.ensureRuntimeException(Base.java:288)
at com.tangosol.run.xml.XmlHelper.loadResourceInternal(XmlHelper.java:34
1)
at com.tangosol.run.xml.XmlHelper.loadFileOrResource(XmlHelper.java:283)
at com.tangosol.io.pof.ConfigurablePofContext.createPofConfig(Configurab
lePofContext.java:835)
at com.tangosol.io.pof.ConfigurablePofContext.initialize(ConfigurablePof
Context.java:797)
at com.tangosol.io.pof.ConfigurablePofContext.setContextClassLoader(Conf
igurablePofContext.java:322)
at com.tangosol.io.ConfigurableSerializerFactory.createSerializer(Config
urableSerializerFactory.java:42)
... 13 more
Caused by: java.io.IOException: The POF configuration is missing: "Mypof-config.
xml", loader=sun.misc.Launcher$AppClassLoader@1b90b39
at com.tangosol.run.xml.XmlHelper.loadResourceInternal(XmlHelper.java:31
8)
... 18 morePlease find the total log on the Cache-server.cmd
Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_ANNOUNCE),
Id=0, Version=3.7.1}
ToMemberSet=null
NotifySent=false
LastRecvTimestamp=none
MemberSet=MemberSet(Size=1, ids=[1])
Terminate batch job (Y/N)? Y
C:\OracleCoherence\coherence\bin>cache-server
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) Client VM (build 24.60-b09, mixed mode, sharing)
2014-08-05 13:25:25.753/0.327 Oracle Coherence 3.7.1.0 <Info> (thread=main, memb
er=n/a): Loaded operational configuration from "jar:file:/C:/OracleCoherence/coh
erence/lib/coherence.jar!/tangosol-coherence.xml"
2014-08-05 13:25:25.816/0.390 Oracle Coherence 3.7.1.0 <Info> (thread=main, memb
er=n/a): Loaded operational overrides from "jar:file:/C:/OracleCoherence/coheren
ce/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2014-08-05 13:25:25.816/0.390 Oracle Coherence 3.7.1.0 <D5> (thread=main, member
=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not
specified
2014-08-05 13:25:25.816/0.390 Oracle Coherence 3.7.1.0 <D5> (thread=main, member
=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.7.1.0 Build 27797
Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
2014-08-05 13:25:25.972/0.546 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, m
ember=n/a): Loaded cache configuration from "file:/C:/OracleCoherence/coherence/
lib/cache-config.xml"; this document does not refer to any schema definition and
has not been validated.
2014-08-05 13:25:26.440/1.014 Oracle Coherence GE 3.7.1.0 <Warning> (thread=main
, member=n/a): Local address "127.0.0.1" is a loopback address; this cluster nod
e will not connect to nodes located on different machines
2014-08-05 13:25:26.830/1.404 Oracle Coherence GE 3.7.1.0 <D4> (thread=main, mem
ber=n/a): TCMP bound to /127.0.0.1:8088 using SystemSocketProvider
2014-08-05 13:25:30.558/5.132 Oracle Coherence GE 3.7.1.0 <Info> (thread=Cluster
, member=n/a): Created a new cluster "cluster:0xFCC1" with Member(Id=1, Timestam
p=2014-08-05 13:25:27.251, Address=127.0.0.1:8088, MachineId=45419, Location=sit
e:,process:18420, Role=CoherenceServer, Edition=Grid Edition, Mode=Development,
CpuCount=2, SocketCount=2) UID=0x7F00000100000147A52A6E93B16B1F98
2014-08-05 13:25:30.558/5.132 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, m
ember=n/a): Started cluster Name=cluster:0xFCC1
Group{Address=224.3.7.0, Port=37000, TTL=0}
MasterMemberSet(
ThisMember=Member(Id=1, Timestamp=2014-08-05 13:25:27.251, Address=127.0.0.1:8
088, MachineId=45419, Location=site:,process:18420, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2014-08-05 13:25:27.251, Address=127.0.0.1
:8088, MachineId=45419, Location=site:,process:18420, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1
Member(Id=1, Timestamp=2014-08-05 13:25:27.251, Address=127.0.0.1:8088, Mach
ineId=45419, Location=site:,process:18420, Role=CoherenceServer)
MemberId|ServiceVersion|ServiceJoined|MemberState
1|3.7.1|2014-08-05 13:25:30.558|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}
2014-08-05 13:25:30.620/5.194 Oracle Coherence GE 3.7.1.0 <Error> (thread=Cluste
r, member=1): StopRunning ClusterService{Name=Cluster, State=(SERVICE_STARTED, S
TATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1} due to unhandled exception:
Exception in thread "main" (Wrapped: Failed to start Service "Management" (Servi
ceState=SERVICE_STOPPED)) java.lang.RuntimeException: Failed to start Service "C
luster" (ServiceState=SERVICE_STOPPED, STATE_JOINED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.s
tart(Service.CDB:38)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.start(Grid.CDB:6)
at com.tangosol.coherence.component.util.SafeService.startService(SafeSe
rvice.CDB:39)
at com.tangosol.coherence.component.util.SafeService.ensureRunningServic
e(SafeService.CDB:27)
at com.tangosol.coherence.component.util.SafeService.start(SafeService.C
DB:14)
at com.tangosol.coherence.component.net.management.Connector.startServic
e(Connector.CDB:58)
at com.tangosol.coherence.component.net.management.gateway.Remote.regist
erLocalModel(Remote.CDB:10)
at com.tangosol.coherence.component.net.management.Gateway.register(Gate
way.CDB:6)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluste
r(SafeCluster.CDB:46)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.C
DB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInterna
l(DefaultConfigurableCacheFactory.java:968)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(Defaul
tConfigurableCacheFactory.java:937)
at com.tangosol.net.DefaultCacheServer.startServices(DefaultCacheServer.
java:81)
at com.tangosol.net.DefaultCacheServer.intialStartServices(DefaultCacheS
erver.java:250)
at com.tangosol.net.DefaultCacheServer.startAndMonitor(DefaultCacheServe
r.java:55)
at com.tangosol.net.DefaultCacheServer.main(DefaultCacheServer.java:197)
Caused by: java.lang.RuntimeException: Failed to start Service "Cluster" (Servic
eState=SERVICE_STOPPED, STATE_JOINED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.w
aitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService.ensureService(ClusterService.CDB:15)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService.doServiceJoining(ClusterService.CDB:47)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onServiceState(Grid.CDB:23)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.s
etServiceState(Service.CDB:8)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.setServiceState(Grid.CDB:21)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid$NotifyStartup.onReceived(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onMessage(Grid.CDB:34)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onNotify(Grid.CDB:33)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Unknown Source)
2014-08-05 13:25:30.620/5.194 Oracle Coherence GE 3.7.1.0 <Error> (thread=Cluste
r, member=1):
(Wrapped: Failed to load POF configuration: C:\OracleCoherence\coherence\lib\Myp
ofconfig.xml) java.io.IOException: The POF configuration is missing: "C:\OracleC
oherence\coherence\lib\Mypofconfig.xml", loader=sun.misc.Launcher$AppClassLoader
@1b90b39
at com.tangosol.util.Base.ensureRuntimeException(Base.java:288)
at com.tangosol.run.xml.XmlHelper.loadResourceInternal(XmlHelper.java:34
1)
at com.tangosol.run.xml.XmlHelper.loadFileOrResource(XmlHelper.java:283)
at com.tangosol.io.pof.ConfigurablePofContext.createPofConfig(Configurab
lePofContext.java:835)
at com.tangosol.io.pof.ConfigurablePofContext.initialize(ConfigurablePof
Context.java:797)
at com.tangosol.io.pof.ConfigurablePofContext.setContextClassLoader(Conf
igurablePofContext.java:322)
at com.tangosol.util.ExternalizableHelper.ensureSerializer(Externalizabl
eHelper.java:291)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.e
nsureSerializer(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.e
nsureSerializer(Service.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.w
riteObject(Service.CDB:1)
at com.tangosol.coherence.component.net.Message.writeObject(Message.CDB:
1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService$ServiceJoining.write(ClusterService.CDB:8)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.serializeMessage(Grid.CDB:14)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetPro
cessor.PacketPublisher.packetizeMessage(PacketPublisher.CDB:17)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetPro
cessor.PacketPublisher$InQueue.add(PacketPublisher.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.dispatchMessage(Grid.CDB:62)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.post(Grid.CDB:31)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.send(Grid.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService$ServiceJoinRequest.proceed(ClusterService.CDB:35)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService.validateNewService(ClusterService.CDB:88)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService$ServiceJoinRequest.onReceived(ClusterService.CDB:66)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onMessage(Grid.CDB:34)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onNotify(Grid.CDB:33)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService.onNotify(ClusterService.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: The POF configuration is missing: "C:\OracleCohe
rence\coherence\lib\Mypofconfig.xml", loader=sun.misc.Launcher$AppClassLoader@1b
90b39
at com.tangosol.run.xml.XmlHelper.loadResourceInternal(XmlHelper.java:31
8)
... 24 more
2014-08-05 13:25:30.620/5.194 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster,
member=n/a): Service Cluster left the cluster
2014-08-05 13:25:30.620/5.194 Oracle Coherence GE 3.7.1.0 <Error> (thread=Invoca
tion:Management, member=n/a): Terminating InvocationService due to unhandled exc
eption: java.lang.RuntimeException
2014-08-05 13:25:30.620/5.194 Oracle Coherence GE 3.7.1.0 <Error> (thread=Invoca
tion:Management, member=n/a):
java.lang.RuntimeException: Failed to start Service "Cluster" (ServiceState=SERV
ICE_STOPPED, STATE_JOINED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.w
aitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService.ensureService(ClusterService.CDB:15)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService.doServiceJoining(ClusterService.CDB:47)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onServiceState(Grid.CDB:23)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.s
etServiceState(Service.CDB:8)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.setServiceState(Grid.CDB:21)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid$NotifyStartup.onReceived(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onMessage(Grid.CDB:34)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onNotify(Grid.CDB:33)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Unknown Source)
2014-08-05 13:25:30.620/5.194 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocatio
n:Management, member=n/a): Service Management left the cluster
2014-08-05 13:25:30.636/5.210 Oracle Coherence GE 3.7.1.0 <Error> (thread=main,
member=n/a): Error while starting service "Management": (Wrapped: Failed to star
t Service "Management" (ServiceState=SERVICE_STOPPED)) java.lang.RuntimeExceptio
n: Failed to start Service "Cluster" (ServiceState=SERVICE_STOPPED, STATE_JOINED
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.s
tart(Service.CDB:38)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.start(Grid.CDB:6)
at com.tangosol.coherence.component.util.SafeService.startService(SafeSe
rvice.CDB:39)
at com.tangosol.coherence.component.util.SafeService.ensureRunningServic
e(SafeService.CDB:27)
at com.tangosol.coherence.component.util.SafeService.start(SafeService.C
DB:14)
at com.tangosol.coherence.component.net.management.Connector.startServic
e(Connector.CDB:58)
at com.tangosol.coherence.component.net.management.gateway.Remote.regist
erLocalModel(Remote.CDB:10)
at com.tangosol.coherence.component.net.management.Gateway.register(Gate
way.CDB:6)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluste
r(SafeCluster.CDB:46)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.C
DB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInterna
l(DefaultConfigurableCacheFactory.java:968)
at com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(Defaul
tConfigurableCacheFactory.java:937)
at com.tangosol.net.DefaultCacheServer.startServices(DefaultCacheServer.
java:81)
at com.tangosol.net.DefaultCacheServer.intialStartServices(DefaultCacheS
erver.java:250)
at com.tangosol.net.DefaultCacheServer.startAndMonitor(DefaultCacheServe
r.java:55)
at com.tangosol.net.DefaultCacheServer.main(DefaultCacheServer.java:197)
Caused by: java.lang.RuntimeException: Failed to start Service "Cluster" (Servic
eState=SERVICE_STOPPED, STATE_JOINED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.w
aitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.poll(Grid.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.poll(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService.ensureService(ClusterService.CDB:15)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.ClusterService.doServiceJoining(ClusterService.CDB:47)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onServiceState(Grid.CDB:23)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.s
etServiceState(Service.CDB:8)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.setServiceState(Grid.CDB:21)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid$NotifyStartup.onReceived(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onMessage(Grid.CDB:34)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onNotify(Grid.CDB:33)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Unknown Source)
C:\OracleCoherence\coherence\bin>
And the Cache-Server content is
@echo off
@rem This will start a cache server
setlocal
:config
@rem specify the Coherence installation directory
set coherence_home=%~dp0\..
@rem specify the JVM heap size
set memory=512m
set java_home=
:start
if not exist "%coherence_home%\lib\coherence.jar" goto instructions
:launch
if "%1"=="-jmx" (
set jmxproperties=-Dcom.sun.management.jmxremote -Dtangosol.coherence.management=all -Dtangosol.coherence.management.remote=true
shift
set java_opts=-Xms%memory% -Xmx%memory% %jmxproperties%
java -server -showversion -Dtangosol.coherence.ttl=0 -Dtangosol.coherence.localhost=127.0.0.1 -Dtangosol.pof.enabled=true -
Dtangosol.coherence.cacheconfig=C:\OracleCoherence\coherence\lib\cache-config.xml -Dtangosol.pof.config=C:\OracleCoherence\coherence\lib\Mypofconfig.xml
%java_opts% -cp "%coherence_home%\lib\coherence.jar" com.tangosol.net.DefaultCacheServer %1
goto exit
:instructions
echo Usage:
echo ^<coherence_home^>\bin\cache-server.cmd
goto exit
:exit
endlocal
@echo on -
Veritas Cluster - what is clusterservice service group?
I am updating a patch VCS MP, per the documentation I see I need to offline clusterService goup by seeing the output of hagrp -status. Out of the bunch of service groups how do I know whcih is clusterservice service group ? Please help.
Please have a look if there is an service group like
VRTSWebApp VCSweb -
Mysterious null pointers during Applet.paint(g)
I'm working with the paint function of my applet, which uses double buffering, and I occassionally get an odd null pointer exception, which does not halt its excution, but I believe may be linked to some other unusual behavior regarding components not being painted correctly.
I've been staring at this thing for a while now and I am completely out of ideas.
For debugging purposes, I've expressly checked each object invovled in the painting to ensure that it isn't null, but the exception is still being thrown and java is telling me that the exception is being trown on line numbers that have only a closing brace (see code)
Any help is much appreciated
Here's the code... state is an extension of Container and loading is just a label to display when I'm changing different containers in and out of state
public void paint(Graphics g) {
if(offscreen == null) offscreen = createImage(500, 480);
else {
Graphics g1 = offscreen.getGraphics();
if(!dontPaint) {
if(loading != null && loading.isVisible()) loading.setVisible(false);
if(state!=null) state.paint(g1);
else {
if(g1 != null) {
g1.setColor(Color.black);
g1.fillRect(0, 0, 500, 480);
if(!loading.isVisible() && loading!=null) loading.setVisible(true);
if(g != null) g.drawImage(offscreen, 0, 0, null);
}here's an example of the errors:
java.lang.NullPointerException
at sun.java2d.pipe.DrawImage.copyImage(DrawImage.java:48)
at sun.java2d.pipe.DrawImage.copyImage(DrawImage.java:715)
at sun.java2d.pipe.ValidatePipe.copyImage(ValidatePipe.java:147)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:2782)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:2772)
at StarDart.paint(StarDart.java:113)
at StarDart.update(StarDart.java:96)
at sun.awt.RepaintArea.paint(RepaintArea.java:172)
at sun.awt.windows.WComponentPeer.handleEvent(WComponentPeer.java:260)
at java.awt.Component.dispatchEventImpl(Component.java:3586)
at java.awt.Container.dispatchEventImpl(Container.java:1437)
at java.awt.Component.dispatchEvent(Component.java:3367)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:445)
at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchTh
read.java:190)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThre
ad.java:144)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:130)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:98)I do have a thread which is always running; I use the Runnable interface. Here is my run() function:
     public void run() {
          while(!stopRunning) {
               for(; sessionInput(); state.handleInput(inBuf));
                      repaint();
                    try {
                              Thread.sleep(100L);
                    } catch(InterruptedException interruptedexception) { System.out.println(interruptedexception); }
          }The for loop manages my networking code. Should I be putting the call to repaint() in a try block?
Does this have something to do with synchronization? -
Managed server not able to join the cluster
Hi
I have two storage node enabled coherence servers on two different machines.These two are able to form the cluster without any problem. I also have two Managed servers. When I start one, will join the cluster without any issue but when I start the fourth one which does not join the cluster. Only one Managed server joins the cluster. I am getting the following error.
2011-12-22 15:39:26.940/356.798 Oracle Coherence GE 3.6.0.4 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded cache configuration from "file:/u02/oracle/admin/atddomain/atdcluster/ATD/config/atd-client-cache-config.xml"
2011-12-22 15:39:26.943/356.801 Oracle Coherence GE 3.6.0.4 <D4> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): TCMP bound to /172.23.34.91:8190 using SystemSocketProvider
2011-12-22 15:39:57.909/387.767 Oracle Coherence GE 3.6.0.4 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2011-12-22 15:39:26.944, Address=172.23.34.91:8190, MachineId=39242, Location=site:dev.icd,machine:appsoad2-web2,process:24613, Role=WeblogicServer) has been attempting to join the cluster at address 231.1.1.50:7777 with TTL 4 for 30 seconds without success; this could indicate a mis-configured TTL value, or it may simply be the result of a busy cluster or active failover.
2011-12-22 15:39:57.909/387.767 Oracle Coherence GE 3.6.0.4 <Warning> (thread=Cluster, member=n/a): Received a discovery message that indicates the presence of an existing cluster:
Message "NewMemberAnnounceWait"
FromMember=Member(Id=2, Timestamp=2011-12-22 15:22:56.607, Address=172.23.34.74:8090, MachineId=39242, Location=site:dev.icd,machine:appsoad4,process:23937,member:CoherenceServer2, Role=WeblogicWeblogicCacheServer)
FromMessageId=0
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=9
ToPollId=0
Poll=null
Packets
[000]=Broadcast{PacketType=0x0DDF00D2, ToId=0, FromId=2, Direction=Incoming, ReceivedMillis=15:39:57.909, MessageType=9, ServiceId=0, MessagePartCount=1, MessagePartIndex=0, Body=0}
Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_ANNOUNCE), Id=0, Version=3.6}
ToMemberSet=null
NotifySent=false
ToMember=Member(Id=0, Timestamp=2011-12-22 15:39:26.944, Address=172.23.34.91:8190, MachineId=39242, Location=site:dev.icd,machine:appsoad2-web2,process:24613, Role=WeblogicServer)
SeniorMember=Member(Id=1, Timestamp=2011-12-22 15:22:53.032, Address=172.23.34.73:8090, MachineId=39241, Location=site:dev.icd,machine:appsoad3,process:19339,member:CoherenceServer1, Role=WeblogicWeblogicCacheServer)
2011-12-22 15:40:02.915/392.773 Oracle Coherence GE 3.6.0.4 <Warning> (thread=Cluster, member=n/a): Received a discovery message that indicates the presence of an existing cluster:
Message "NewMemberAnnounceWait"
FromMember=Member(Id=2, Timestamp=2011-12-22 15:22:56.607, Address=172.23.34.74:8090, MachineId=39242, Location=site:dev.icd,machine:appsoad4,process:23937,member:CoherenceServer2, Role=WeblogicWeblogicCacheServer)
FromMessageId=0
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=9
ToPollId=0
Poll=null
Packets
{Hi,
By default Coherence uses a multicast protocol to discover other nodes when forming a cluster. Since you are having difficulties in establishing a cluster via multicast, Can you please perform a multicast test and see if multicast is configured properly.
http://wiki.tangosol.com/display/COH32UG/Multicast+Test
Hope you are using same configuration files across the cluster members and all members of the cluster must specify the same cluster name in order to be allowed to join the cluster.
<cluster-name system-property="tangosol.coherence.cluster";>xxx</cluster-name>
I would suggest, try using the unicast-listener with the well-known-addresses instead of muticast-listener.
http://wiki.tangosol.com/display/COH32UG/well-known-addresses
Add similar entries like below in your tangosol override xml..
<well-known-addresses>
<socket-address id="1">
<address> 172.23.34.91<;/address>
<port>8190</port>
</socket-address>
<socket-address id="2">
<address> 172.23.34.74<;/address>
<port> 8090</port>
</socket-address>
</well-known-addresses>
This list is used by all other nodes to find their way into the cluster without the use of multicast, thus at least one well known node must be running for other nodes to be able to join.
Hope this helps!!
Thanks,
Ashok.
<div id="isChromeWebToolbarDiv" style="display:none"></div> -
Wrapped: Unable to load class "com.cache.DBCacheStore"
Hi,
I was trying out a sample application for using DBCacheStore and getting the following error...
(Wrapped: Unable to load class "com.cache.DBCacheStore" using sun.misc.Launc
her$AppClassLoader@13f5d07
<class-scheme>
<class-name>com.cache.DBCacheStore</class-name>
</class-scheme>) java.lang.ClassNotFoundException: com.cache.DBCacheStore
I have the CLASSPATH set to the location of DBCacheStore.class file both in cache-server.cmd and in my application startup JVM params. But still i am not able to resolve this issue. However i am still able to hit the database and get the values for the first call and on sub-sequent call, i am able to retrive the value from cache. I also encounter the below error, after i have retrieved the values from cache...
2010-11-10 15:08:31.553/29.671 Oracle Coherence GE 3.6.0.0 <Error> (thread=Distr
ibutedCache, member=1): Terminating PartitionedCache due to unhandled exception:
java.lang.UnsupportedOperationException
2010-11-10 15:08:31.553/29.671 Oracle Coherence GE 3.6.0.0 <Error> (thread=Distr
ibutedCache, member=1):
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableMap.put(Collections.java:1285)
As the exception occurs, the cluster gets restarted and when i re-run the application, it again goes for a DB hit instead of cache hit.
Note : I have the cache-server.cmd running in one the console window.
Following is my cache-config.xml file content...
<?xml version="1.0" encoding="UTF-8" ?>
<cache-config>
<caching-scheme-mapping>
<!--
Caches with names that start with 'Virtual' will be created
as distributed-db-backed.
-->
<cache-mapping>
<cache-name>Virtual*</cache-name>
<scheme-name>distributed-db-backed</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<!--
DB Backed Distributed caching scheme.
-->
<distributed-scheme>
<scheme-name>distributed-db-backed</scheme-name>
<service-name>DistributedCache</service-name>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>
com.tangosol.util.ObservableHashMap
</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>
com.cache.DBCacheStore
</class-name>
<init-params>
<init-param>
<param-type>
java.lang.String
</param-type>
<param-value>
EMP_ADDR_VIEW
</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<read-only>false</read-only>
<!--
To make this a write-through cache just change the value below to 0 (zero)
-->
<write-delay-seconds>0</write-delay-seconds>
</read-write-backing-map-scheme>
</backing-map-scheme>
<listener />
<autostart>true</autostart>
</distributed-scheme>
</caching-schemes>
</cache-config>
Thanks
Karthik
Edited by: user1073902 on Nov 10, 2010 4:25 AMHello Noah,
I have listed the contents of log file ( for both cache-server.cmd and the client application ).
cache-server.cmd Log File
=========================
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) Server VM (build 16.3-b01, mixed mode)
2010-11-11 09:51:49.634/0.236 Oracle Coherence 3.6.0.0 <Info> (thread=main, memb
er=n/a): Loaded operational configuration from "jar:file:/E:/coherence/lib/coher
ence.jar!/tangosol-coherence.xml"
2010-11-11 09:51:49.634/0.236 Oracle Coherence 3.6.0.0 <Info> (thread=main, memb
er=n/a): Loaded operational overrides from "jar:file:/E:/coherence/lib/coherence
.jar!/tangosol-coherence-override-dev.xml"
2010-11-11 09:51:49.634/0.236 Oracle Coherence 3.6.0.0 <D5> (thread=main, member
=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not
specified
2010-11-11 09:51:49.649/0.251 Oracle Coherence 3.6.0.0 <D5> (thread=main, member
=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
2010-11-11 09:51:49.649/0.251 Oracle Coherence 3.6.0.0 <D6> (thread=main, member
=n/a): Loaded edition data from "jar:file:/E:/coherence/lib/coherence.jar!/coher
ence-grid.xml"
Oracle Coherence Version 3.6.0.0 Build 17229
Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
2010-11-11 09:51:49.885/0.487 Oracle Coherence GE 3.6.0.0 <Info> (thread=main, m
ember=n/a): Loaded cache configuration from "file:/E:/bea/user_projects/workspac
es/workSpaceStudio/coh/cache-config.xml"
2010-11-11 09:51:50.231/0.833 Oracle Coherence GE 3.6.0.0 <D4> (thread=main, mem
ber=n/a): TCMP bound to /10.144.32.78:8088 using SystemSocketProvider
2010-11-11 09:51:53.786/4.388 Oracle Coherence GE 3.6.0.0 <Info> (thread=Cluster
, member=n/a): Created a new cluster "mycluster" with Member(Id=1, Timestamp=201
0-11-11 09:51:50.231, Address=10.144.32.78:8088, MachineId=47694, Location=site:
XyzT.CORP.Xyz.IN,machine:hdchshocms2026,process:4472, Role=CoherenceServer, Edit
ion=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) UID=0x0A90204E000
0012C392B0817BA4E1F98
2010-11-11 09:51:53.786/4.388 Oracle Coherence GE 3.6.0.0 <Info> (thread=main, m
ember=n/a): Started cluster Name=mycluster
Group{Address=224.0.0.1, Port=5455, TTL=4}
MasterMemberSet
ThisMember=Member(Id=1, Timestamp=2010-11-11 09:51:50.231, Address=10.144.32.7
8:8088, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,p
rocess:4472, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2010-11-11 09:51:50.231, Address=10.144.32
.78:8088, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026
,process:4472, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2010-11-11 09:51:50.231, Address=10.144.32.78:8088, M
achineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,process:44
72, Role=CoherenceServer)
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0, BitSetCount=0
TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}
2010-11-11 09:51:53.833/4.435 Oracle Coherence GE 3.6.0.0 <D5> (thread=Invocatio
n:Management, member=1): Service Management joined the cluster with senior servi
ce member 1
2010-11-11 09:51:54.085/4.687 Oracle Coherence GE 3.6.0.0 <D5> (thread=Distribut
edCache, member=1): Service DistributedCache joined the cluster with senior serv
ice member 1
2010-11-11 09:51:54.101/4.703 Oracle Coherence GE 3.6.0.0 <D6> (thread=Distribut
edCache, member=1): Service DistributedCache: sending PartitionConfig ConfigSync
to all
2010-11-11 09:51:54.116/4.718 Oracle Coherence GE 3.6.0.0 <Info> (thread=main, m
ember=1):
Services
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Vers
ion=3.6, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1,
OldestMemberId=1}
PartitionedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=
enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartit
ions=0}
Started DefaultCacheServer...
2010-11-11 09:55:41.209/231.811 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster
, member=1): Member(Id=2, Timestamp=2010-11-11 09:55:41.216, Address=10.144.32.7
8:8090, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,p
rocess:5304, Role=XyzCacheDatabaseCache) joined Cluster with senior member 1
2010-11-11 09:55:41.209/231.811 Oracle Coherence GE 3.6.0.0 <D6> (thread=Cluster
, member=1): TcpRing connecting to Member(Id=2, Timestamp=2010-11-11 09:55:41.21
6, Address=10.144.32.78:8090, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,ma
chine:hdchshocms2026,process:5304, Role=XyzCacheDatabaseCache)
2010-11-11 09:55:41.209/231.811 Oracle Coherence GE 3.6.0.0 <D6> (thread=Cluster
, member=1): TcpRing connected to Member(Id=2, Timestamp=2010-11-11 09:55:41.216
, Address=10.144.32.78:8090, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,mac
hine:hdchshocms2026,process:5304, Role=XyzCacheDatabaseCache)
2010-11-11 09:55:41.287/231.889 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster
, member=1): Member 2 joined Service Management with senior member 1
2010-11-11 09:55:41.553/232.155 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster
, member=1): Member 2 joined Service DistributedCache with senior member 1
2010-11-11 09:55:41.553/232.155 Oracle Coherence GE 3.6.0.0 <Error> (thread=Dist
ributedCache, member=1): BackingMapManager com.tangosol.net.DefaultConfigurableC
acheFactory$Manager: failed to instantiate a cache: VirtualCache
2010-11-11 09:55:41.553/232.155 Oracle Coherence GE 3.6.0.0 <Error> (thread=Dist
ributedCache, member=1):
(Wrapped: Unable to load class "com.cache.DBCacheStore" using sun.misc.Launc
her$AppClassLoader@13f5d07
<class-scheme>
<class-name>com.cache.DBCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>EMP_ADDR_VIEW</param-value>
</init-param>
</init-params>
</class-scheme>) java.lang.ClassNotFoundException: com.cache.DBCacheStore
at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
at com.tangosol.run.xml.XmlHelper.createInstance(XmlHelper.java:2487)
at com.tangosol.net.DefaultConfigurableCacheFactory.instantiateAny(Defau
ltConfigurableCacheFactory.java:3256)
at com.tangosol.net.DefaultConfigurableCacheFactory.instantiateCacheStor
e(DefaultConfigurableCacheFactory.java:3106)
at com.tangosol.net.DefaultConfigurableCacheFactory.instantiateReadWrite
BackingMap(DefaultConfigurableCacheFactory.java:1674)
at com.tangosol.net.DefaultConfigurableCacheFactory.configureBackingMap(
DefaultConfigurableCacheFactory.java:1429)
at com.tangosol.net.DefaultConfigurableCacheFactory$Manager.instantiateB
ackingMap(DefaultConfigurableCacheFactory.java:3904)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.partitionedService.PartitionedCache$Storage.instantiateResourceMap(Partition
edCache.CDB:21)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.partitionedService.PartitionedCache$Storage.setCacheName(PartitionedCache.CD
B:25)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.partitionedService.PartitionedCache$ServiceConfig$ConfigListener.entryInsert
ed(PartitionedCache.CDB:17)
at com.tangosol.util.MapEvent.dispatch(MapEvent.java:266)
at com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
at com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.jav
a:556)
at com.tangosol.util.ObservableHashMap.dispatchEvent(ObservableHashMap.j
ava:229)
at com.tangosol.util.ObservableHashMap$Entry.onAdd(ObservableHashMap.jav
a:270)
at com.tangosol.util.SafeHashMap.put(SafeHashMap.java:244)
at com.tangosol.coherence.component.util.ServiceConfig$Map.put(ServiceCo
nfig.CDB:43)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.partitionedService.PartitionedCache$StorageIdRequest.onReceived(PartitionedC
ache.CDB:45)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onMessage(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onNotify(Grid.CDB:33)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.PartitionedService.onNotify(PartitionedService.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException: com.cache.DBCacheStore
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at com.tangosol.util.ExternalizableHelper.loadClass(ExternalizableHelper
.java:3011)
at com.tangosol.run.xml.XmlHelper.createInstance(XmlHelper.java:2421)
... 22 more
2010-11-11 09:55:41.631/232.233 Oracle Coherence GE 3.6.0.0 <D5> (thread=Distrib
utedCache, member=1): 3> Transferring primary PartitionSet{0, 1, 2, 3, 4, 5, 6,
7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,
68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87,
88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 1
06, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 1
22, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 1
38, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 1
54, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 1
70, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 1
86, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 2
02, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 2
18, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 2
34, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 2
50, 251, 252, 253, 254, 255, 256} to member 2 requesting 128
2010-11-11 09:55:41.741/232.343 Oracle Coherence GE 3.6.0.0 <D4> (thread=Distrib
utedCache, member=1): 1> Transferring 129 out of 129 partitions to a node-safe b
ackup 1 at member 2 (under 129)
2010-11-11 09:55:41.757/232.359 Oracle Coherence GE 3.6.0.0 <D5> (thread=Distrib
utedCache, member=1): Transferring 0KB of backup[1] for PartitionSet{128, 129, 1
30, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 1
46, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 1
62, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 1
78, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 1
94, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 2
10, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 2
26, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 2
42, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256} to mem
ber 2
2010-11-11 09:55:42.070/232.672 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster
, member=1): TcpRing disconnected from Member(Id=2, Timestamp=2010-11-11 09:55:4
1.216, Address=10.144.32.78:8090, MachineId=47694, Location=site:XyzT.CORP.Xyz.I
N,machine:hdchshocms2026,process:5304, Role=XyzCacheDatabaseCache) due to a peer
departure; removing the member.
2010-11-11 09:55:42.070/232.672 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster
, member=1): Member 2 left service Management with senior member 1
2010-11-11 09:55:42.070/232.672 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster
, member=1): Member 2 left service DistributedCache with senior member 1
2010-11-11 09:55:42.070/232.672 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster
, member=1): Member(Id=2, Timestamp=2010-11-11 09:55:42.07, Address=10.144.32.78
:8090, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,pr
ocess:5304, Role=XyzCacheDatabaseCache) left Cluster with senior member 1
2010-11-11 09:55:42.101/232.703 Oracle Coherence GE 3.6.0.0 <Error> (thread=Dist
ributedCache, member=1): Terminating PartitionedCache due to unhandled exception
: java.lang.UnsupportedOperationException
2010-11-11 09:55:42.101/232.703 Oracle Coherence GE 3.6.0.0 <Error> (thread=Dist
ributedCache, member=1):
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableMap.put(Collections.java:1285)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.partitionedService.PartitionedCache$Storage.putPrimaryResource(PartitionedCa
che.CDB:44)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.partitionedService.PartitionedCache$Storage.moveResourcesToPrimary(Partition
edCache.CDB:50)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.partitionedService.PartitionedCache$Storage.movePartition(PartitionedCache.C
DB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.partitionedService.PartitionedCache.movePartition(PartitionedCache.CDB:14)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.PartitionedService.assignPrimaryPartition(PartitionedService.CDB:38)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.PartitionedService.restoreOrphans(PartitionedService.CDB:42)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.PartitionedService.onOwnershipRequest(PartitionedService.CDB:13)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.PartitionedService$OwnershipRequest.onReceived(PartitionedService.CDB:5)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onMessage(Grid.CDB:11)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.G
rid.onNotify(Grid.CDB:33)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.PartitionedService.onNotify(PartitionedService.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.g
rid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:619)
2010-11-11 09:55:42.101/232.703 Oracle Coherence GE 3.6.0.0 <D5> (thread=Distrib
utedCache, member=1): Service DistributedCache left the cluster
2010-11-11 09:55:45.012/235.614 Oracle Coherence GE 3.6.0.0 <Info> (thread=main,
member=1): Restarting Service: DistributedCache
2010-11-11 09:55:45.012/235.614 Oracle Coherence GE 3.6.0.0 <D5> (thread=Distrib
utedCache, member=1): Service DistributedCache joined the cluster with senior se
rvice member 1
2010-11-11 09:55:45.012/235.614 Oracle Coherence GE 3.6.0.0 <D6> (thread=Distrib
utedCache, member=1): Service DistributedCache: sending PartitionConfig ConfigSy
nc to all
Client Application Log
======================
2010-11-11 09:59:07.380/0.251 Oracle Coherence 3.6.0.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/E:/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2010-11-11 09:59:07.380/0.251 Oracle Coherence 3.6.0.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/E:/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2010-11-11 09:59:07.380/0.251 Oracle Coherence 3.6.0.0 <D5> (thread=main, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
2010-11-11 09:59:07.380/0.251 Oracle Coherence 3.6.0.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
2010-11-11 09:59:07.395/0.266 Oracle Coherence 3.6.0.0 <D6> (thread=main, member=n/a): Loaded edition data from "jar:file:/E:/coherence/lib/coherence.jar!/coherence-grid.xml"
Oracle Coherence Version 3.6.0.0 Build 17229
Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
2010-11-11 09:59:07.896/0.767 Oracle Coherence GE 3.6.0.0 <D4> (thread=main, member=n/a): TCMP bound to /10.144.32.78:8090 using SystemSocketProvider
2010-11-11 09:59:09.164/2.035 Oracle Coherence GE 3.6.0.0 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Timestamp=2010-11-11 09:59:09.164, Address=10.144.32.78:8090, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,process:2404, Role=XyzCacheDatabaseCache, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) joined cluster "mycluster" with senior Member(Id=1, Timestamp=2010-11-11 09:58:30.832, Address=10.144.32.78:8088, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,process:4772, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
2010-11-11 09:59:09.180/2.051 Oracle Coherence GE 3.6.0.0 <D6> (thread=Cluster, member=n/a): TcpRing connecting to Member(Id=1, Timestamp=2010-11-11 09:58:30.832, Address=10.144.32.78:8088, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,process:4772, Role=CoherenceServer)
2010-11-11 09:59:09.180/2.051 Oracle Coherence GE 3.6.0.0 <D6> (thread=Cluster, member=n/a): TcpRing connected to Member(Id=1, Timestamp=2010-11-11 09:58:30.832, Address=10.144.32.78:8088, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,process:4772, Role=CoherenceServer)
2010-11-11 09:59:09.180/2.051 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Cluster with senior member 1
2010-11-11 09:59:09.180/2.051 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 1
2010-11-11 09:59:09.180/2.051 Oracle Coherence GE 3.6.0.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service DistributedCache with senior member 1
2010-11-11 09:59:09.180/2.051 Oracle Coherence GE 3.6.0.0 <Info> (thread=main, member=n/a): Started cluster Name=mycluster
Group{Address=224.0.0.1, Port=5455, TTL=4}
MasterMemberSet
ThisMember=Member(Id=2, Timestamp=2010-11-11 09:59:09.164, Address=10.144.32.78:8090, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,process:2404, Role=XyzCacheDatabaseCache)
OldestMember=Member(Id=1, Timestamp=2010-11-11 09:58:30.832, Address=10.144.32.78:8088, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,process:4772, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=2, BitSetCount=2
Member(Id=1, Timestamp=2010-11-11 09:58:30.832, Address=10.144.32.78:8088, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,process:4772, Role=CoherenceServer)
Member(Id=2, Timestamp=2010-11-11 09:59:09.164, Address=10.144.32.78:8090, MachineId=47694, Location=site:XyzT.CORP.Xyz.IN,machine:hdchshocms2026,process:2404, Role=XyzCacheDatabaseCache)
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0, BitSetCount=0
TcpRing{Connections=[1]}
IpMonitor{AddressListSize=0}
2010-11-11 09:59:09.227/2.098 Oracle Coherence GE 3.6.0.0 <D5> (thread=Invocation:Management, member=2): Service Management joined the cluster with senior service member 1
2010-11-11 09:59:09.289/2.160 Oracle Coherence GE 3.6.0.0 <Info> (thread=main, member=2): Loaded cache configuration from "file:/E:/bea/user_projects/workspaces/workSpaceStudio/coh/cache-config.xml"
2010-11-11 09:59:09.461/2.332 Oracle Coherence GE 3.6.0.0 <D5> (thread=DistributedCache, member=2): Service DistributedCache joined the cluster with senior service member 1
DB Hit
DBCacheStore(String) called...
2010-11-11 09:59:09.524/2.395 Oracle Coherence GE 3.6.0.0 <D4> (thread=DistributedCache, member=2): Asking member 1 for 128 primary partitions
load() called...
Emp Id:1, Emp Name:Employee, Emp Addr:Address
Cache Hit
Emp Id:1, Emp Name:Employee, Emp Addr:Address
Client Java Program
===================
package com.cache;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.NamedCache;
public class DatabaseCache {
public DatabaseCache() { }
private NamedCache cache;
public void createCache() {
CacheFactory.ensureCluster();
cache = CacheFactory.getCache("VirtualCache");
public void retrieveEntry() {
System.out.println("DB Hit");
System.out.println(cache.get("1"));
System.out.println("Cache Hit");
System.out.println(cache.get("1"));
public static void main(String[] args) {
DatabaseCache databaseCache = new DatabaseCache();
databaseCache.createCache();
databaseCache.retrieveEntry();
DBCacheStore Java Program
=========================
Note : I am querying an ORACLE VIEW
package com.cache;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.Collection;
import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import com.tangosol.net.cache.CacheStore;
import com.tangosol.util.Base;
public class DBCacheStore extends Base implements CacheStore {
private String tableName = null;
public DBCacheStore() {
System.out.println("DBCacheStore() called...");
public DBCacheStore(String tableName) {
System.out.println("DBCacheStore(String) called...");
this.tableName = tableName;
private Connection getConnection() throws Exception {
Class.forName("oracle.jdbc.driver.OracleDriver");
String url = "jdbc:oracle:thin:@localhost:1521:xe";
Connection connection = DriverManager.getConnection(url,"system","manager");
return connection;
public void erase(Object arg0) {
System.out.println("erase() called...");
public void eraseAll(Collection arg0) {
System.out.println("eraseAll() called...");
public void store(Object key, Object value) {
System.out.println("store() called...");
public void storeAll(Map arg0) {
System.out.println("storeAll() called...");
public Object load(Object obj) {
System.out.println("load() called...");
Employee employee = null;
Connection con = null;
PreparedStatement pStmt = null;
ResultSet rSet = null;
try {
con = getConnection();
pStmt = con.prepareStatement("select emp_id, emp_name, address from " + tableName + " where emp_id = ?");
pStmt.setInt(1, Integer.parseInt((String)obj));
rSet = pStmt.executeQuery();
if(rSet.next()) {
employee = new Employee();
employee.setEmpId(rSet.getInt("emp_id"));
employee.setEmpName(rSet.getString("emp_name"));
employee.setEmpAddr(rSet.getString("address"));
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
if(rSet != null)
rSet.close();
if(pStmt != null)
pStmt.close();
if(con != null)
con.close();
} catch(Exception e) {
e.printStackTrace();
return employee;
public Map loadAll(Collection arg0) {
System.out.println("loadAll() called...");
Connection con = null;
PreparedStatement pStmt = null;
ResultSet rSet = null;
Employee employee = null;
Map empMap = new HashMap();
try {
con = getConnection();
pStmt = con.prepareStatement("select emp_id, emp_name, address from " + tableName);
rSet = pStmt.executeQuery();
while(rSet.next()) {
employee = new Employee();
employee.setEmpId(rSet.getInt("emp_id"));
employee.setEmpName(rSet.getString("emp_name"));
employee.setEmpAddr(rSet.getString("address"));
empMap.put(employee.getEmpId(), employee);
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
if(rSet != null)
rSet.close();
if(pStmt != null)
pStmt.close();
if(con != null)
con.close();
} catch(Exception e) {
e.printStackTrace();
return empMap;
public Iterator keys() {
Connection con = null;
PreparedStatement pStmt = null;
ResultSet rSet = null;
List returnList = null;
try {
con = getConnection();
pStmt = con.prepareStatement("select emp_id from emp_addr_view");
rSet = pStmt.executeQuery();
returnList = new LinkedList();
while(rSet.next()) {
returnList.add(rSet.getString("empId"));
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
if(rSet != null)
rSet.close();
if(pStmt != null)
pStmt.close();
if(con != null)
con.close();
} catch(Exception e) {
e.printStackTrace();
return returnList.iterator();
cache-server.cmd added lines
============================
set CLASSPATH=%CLASSPATH%;E:\bea\user_projects\workspaces\workSpaceStudio\coh\bin;E:\bea\wlserver_10.0\server\lib\ojdbc14.jar;
Note : E:\bea\user_projects\workspaces\workSpaceStudio\coh\bin in the above line contains compiled class files starting from "com" folder.
"%java_exec%" -server -showversion "%java_opts%" -cp E:\bea\user_projects\workspaces\workSpaceStudio\coh\bin -Dtangosol.coherence.log.level=9 -Dtangosol.coherence.cluster=mycluster -Dtangosol.coherence.clusteraddress=224.0.0.1 -Dtangosol.coherence.clusterport=5455 -Dtangosol.coherence.cacheconfig=E:\bea\user_projects\workspaces\workSpaceStudio\coh\cache-config.xml -cp "%coherence_home%\lib\coherence.jar" com.tangosol.net.DefaultCacheServer %1
JVM arguments for running Client Program : DatabaseCache.java
=======================================
-Dtangosol.coherence.cacheconfig=E:\bea\user_projects\workspaces\workSpaceStudio\coh\cache-config.xml
-Dtangosol.coherence.cluster=mycluster
-Dtangosol.coherence.clusteraddress=224.0.0.1
-Dtangosol.coherence.clusterport=5455
==================================================================================================
Please do let me know if you require further information.
Thanks
Karthik -
Overriding eventdistributionpattern-pof throwing unable to load class error
Hi,
We were trying to override the "coherence-eventdistributionpattern-pof-config.xml" with our own event distribution pof due to conflicintg type-id's. We changed type-id's for few of the classes. We changed the type-id for the following classes and left others as such.
com.oracle.coherence.patterns.eventdistribution.EventDistributor$Identifier
com.oracle.coherence.patterns.eventdistribution.events.DistributableEntry
com.oracle.coherence.patterns.eventdistribution.events.DistributableEntryInsertedEvent
com.oracle.coherence.patterns.eventdistribution.events.DistributableEntryUpdatedEvent
com.oracle.coherence.patterns.eventdistribution.events.DistributableEntryRemovedEvent
When i start the coherence server, i get error in loading class "com.oracle.coherence.patterns.eventdistribution.channels.cache.ParallelLocalCacheEventChannelBuilder" for which we did not change the typ-id at all.
Coherence version 3.7.10
Error Trace:
(thread=DistributedCache:DistributedCacheForSequenceGenerators, member=2): PartitionedCache caught an unhandled exception (com.tangosol.util.WrapperException: (Wrapped: error creating class "com.tangosol.io.pof.ConfigurablePofContext") (Wrapped: Unable to load class for user type (Config=test/coherence/grid/obj-pof-config.xml, Type-Id=13402, Class-Name=com.oracle.coherence.patterns.eventdistribution.channels.cache.ParallelLocalCacheEventChannelBuilder)) (Wrapped) com.oracle.coherence.patterns.eventdistribution.channels.cache.ParallelLocalCacheEventChannelBuilder) while exiting.
<Error> (thread=DistributedCache:DistributedCacheForSequenceGenerators, member=2): ClusterService.doServiceLeft: Unknown Service PartitionedCache{Name=DistributedCacheForSequenceGenerators, State=(SERVICE_STOPPED), Not initialized}
(thread=DistributedCache:DistributedCacheForSequenceGenerators, member=2): Service DistributedCacheForSequenceGenerators left the cluster
<Error> (thread=main, member=2): Error while starting service "DistributedCacheForSequenceGenerators": (Wrapped) (Wrapped: error creating class "com.tangosol.io.pof.ConfigurablePofContext") (Wrapped: Unable to load class for user type (Config=test/coherence/grid/obj-pof-config.xml, Type-Id=13402, Class-Name=com.oracle.coherence.patterns.eventdistribution.channels.cache.ParallelLocalCacheEventChannelBuilder)) (Wrapped) java.lang.ClassNotFoundException: com.oracle.coherence.patterns.eventdistribution.channels.cache.ParallelLocalCacheEventChannelBuilder
at com.tangosol.coherence.component.util.Daemon.start(Daemon.CDB:52)
Thanks,File's permissions?
-
High CPU usage with Coherence 3.5.0 cluster node
We're noticing sustained high CPU usage on one of our Coherence nodes - this happens after running a job that pretty intensively interacts with the cache. The jobs complete, however we still see the Coherence process using approximately the equivalent of one core's worth of CPU. I took a couple of thread dumps about fifteen minutes apart, and the only noticeable differences are within the PacketListener1 and PacketListenerN threads - they're holding locks on different DatagramPacket instances, which suggests to be that this may be the piece of code that is looping.
There are two other cache nodes in this particular cluster - both of them on a different machine - and neither of them are exhibiting the same CPU utilization.
Environment:
Coherence 3.5.0
Java - BEA JRockit(R) (build R27.6.3-40_o-112056-1.6.0_11-20090318-2103-linux-x86_64, compiled mode)
Has anyone encountered this scenario before?
Here's a copy of the two thread dumps:
Thread Dump 1 -
===== FULL THREAD DUMP ===============
Thu Feb 24 21:45:00 2011
BEA JRockit(R) R27.6.3-40_o-112056-1.6.0_11-20090318-2103-linux-x86_64
"Main Thread" id=1 idx=0x4 tid=18143 prio=5 alive, in native, waiting
-- Waiting for notification on: java/lang/Class@0x43587b58[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/net/DefaultCacheServer.main(DefaultCacheServer.java:80)
^-- Lock released while waiting: java/lang/Class@0x43587b58[fat lock]
at com/zzzz/carbon/cacheserver/ZzzzCoherenceServerStartup.doWork(ZzzzCoherenceServerStartup.java:29)
at com/zzzz/util/runner/ZzzzRunnerBase.run(ZzzzRunnerBase.java:23)
at com/zzzz/carbon/cacheserver/ZzzzCoherenceServerStartup.main(ZzzzCoherenceServerStartup.java:16)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x8 tid=18144 prio=5 alive, in native, daemon
"(GC Main Thread)" id=3 idx=0xc tid=18145 prio=5 alive, in native, daemon
"(GC Worker Thread 1)" id=? idx=0x10 tid=18146 prio=5 alive, in native, daemon
"(GC Worker Thread 2)" id=? idx=0x14 tid=18147 prio=5 alive, in native, daemon
"(GC Worker Thread 3)" id=? idx=0x18 tid=18148 prio=5 alive, in native, daemon
"(GC Worker Thread 4)" id=? idx=0x1c tid=18149 prio=5 alive, in native, daemon
"(Code Generation Thread 1)" id=4 idx=0x20 tid=18150 prio=5 alive, in native, native_waiting, daemon
"(Code Optimization Thread 1)" id=5 idx=0x24 tid=18151 prio=5 alive, in native, native_waiting, daemon
"(VM Periodic Task)" id=6 idx=0x28 tid=18152 prio=10 alive, in native, daemon
"Finalizer" id=7 idx=0x2c tid=18153 prio=8 alive, in native, native_waiting, daemon
at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)
at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
at jrockit/memory/Finalizer$4.run(Finalizer.java:159)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Reference Handler" id=8 idx=0x30 tid=18154 prio=10 alive, in native, native_waiting, daemon
at java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native Method)
at java/lang/ref/Reference.access$100(Reference.java:11)
at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(Sensor Event Thread)" id=9 idx=0x34 tid=18155 prio=5 alive, in native, daemon
"Logger@9217551 3.5/459" id=12 idx=0x38 tid=18158 prio=3 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/application/console/Coherence$Logger$Queue@0x4e1faef8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/application/console/Coherence$Logger$Queue@0x4e1faef8[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"PacketListener1" id=14 idx=0x3c tid=18159 prio=8 alive, in native, daemon
at java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native Method)
^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb21360[recursive]
at java/net/PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)[inlined]
at java/net/DatagramSocket.receive(DatagramSocket.java:712)[optimized]
^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb21360[thin lock]
^-- Holding lock: java/net/DatagramPacket@0x73aa4e80[thin lock]
^-- Holding lock: java/net/DatagramSocket@0x4cb21338[thin lock]
at com/tangosol/coherence/component/net/socket/UdpSocket.receive(UdpSocket.CDB:20)[optimized]
at com/tangosol/coherence/component/net/UdpPacket.receive(UdpPacket.CDB:4)[optimized]
at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketListener.onNotify(PacketListener.CDB:19)
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"PacketReceiver" id=15 idx=0x40 tid=18160 prio=7 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketReceiver$InQueue@0x4cf60d58[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketReceiver.onWait(PacketReceiver.CDB:2)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketReceiver$InQueue@0x4cf60d58[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"PacketPublisher" id=16 idx=0x44 tid=18161 prio=6 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketPublisher.onWait(PacketPublisher.CDB:2)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"PacketSpeaker" id=17 idx=0x48 tid=18162 prio=8 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketSpeaker$BundlingQueue@0x4cabe9d8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/queue/ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketSpeaker$BundlingQueue@0x4cabe9d8[fat lock]
at com/tangosol/coherence/component/util/queue/ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
at com/tangosol/coherence/component/util/Queue.remove(Queue.CDB:1)
at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketSpeaker.onNotify(PacketSpeaker.CDB:62)
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"PacketListenerN" id=18 idx=0x4c tid=18163 prio=8 alive, in native, daemon
at java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native Method)
^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb221b0[recursive]
at java/net/PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)[inlined]
at java/net/DatagramSocket.receive(DatagramSocket.java:712)[optimized]
^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb221b0[thin lock]
^-- Holding lock: java/net/DatagramPacket@0x73e59890[thin lock]
^-- Holding lock: java/net/MulticastSocket@0x4cb22178[thin lock]
at com/tangosol/coherence/component/net/socket/UdpSocket.receive(UdpSocket.CDB:20)[optimized]
at com/tangosol/coherence/component/net/UdpPacket.receive(UdpPacket.CDB:4)[optimized]
at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketListener.onNotify(PacketListener.CDB:19)
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Cluster|Member(Id=2, Timestamp=2011-02-23 16:25:36.488, Address=10.10.100.77:8095, MachineId=25677, Location=site:emtzzzz.com,machine:njdev04,process:18142,member:cldev201~carboncachelauncher~181)" id=19 idx=0x50 tid=18164 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4cac64d8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4cac64d8[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"TcpRingListener" id=20 idx=0x54 tid=18165 prio=6 alive, in native, daemon
at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
^-- Holding lock: java/net/SocksSocketImpl@0x4cb227d8[thin lock]
at java/net/ServerSocket.implAccept(ServerSocket.java:453)
at java/net/ServerSocket.accept(ServerSocket.java:421)
at com/tangosol/coherence/component/net/socket/TcpSocketAccepter.accept(TcpSocketAccepter.CDB:18)
at com/tangosol/coherence/component/util/daemon/TcpRingListener.acceptConnection(TcpRingListener.CDB:10)
at com/tangosol/coherence/component/util/daemon/TcpRingListener.onNotify(TcpRingListener.CDB:9)
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Invocation:Management:EventDispatcher" id=22 idx=0x58 tid=18166 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher$Queue@0x4ab76140[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher$Queue@0x4ab76140[fat lock]
at com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher.onWait(Service.CDB:7)
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Invocation:Management" id=23 idx=0x5c tid=18167 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4ab75508[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4ab75508[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"DistributedCache" id=25 idx=0x60 tid=18168 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4a3869b0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4a3869b0[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
===== END OF THREAD DUMP ===============
Thread Dump 2 -
===== FULL THREAD DUMP ===============
Thu Feb 24 21:27:53 2011
BEA JRockit(R) R27.6.3-40_o-112056-1.6.0_11-20090318-2103-linux-x86_64
"Main Thread" id=1 idx=0x4 tid=18143 prio=5 alive, in native, waiting
-- Waiting for notification on: java/lang/Class@0x43587b58[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/net/DefaultCacheServer.main(DefaultCacheServer.java:80)
^-- Lock released while waiting: java/lang/Class@0x43587b58[fat lock]
at com/zzzz/carbon/cacheserver/ZzzzCoherenceServerStartup.doWork(ZzzzCoherenceServerStartup.java:29)
at com/zzzz/util/runner/ZzzzRunnerBase.run(ZzzzRunnerBase.java:23)
at com/zzzz/carbon/cacheserver/ZzzzCoherenceServerStartup.main(ZzzzCoherenceServerStartup.java:16)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x8 tid=18144 prio=5 alive, in native, daemon
"(GC Main Thread)" id=3 idx=0xc tid=18145 prio=5 alive, in native, native_waiting, daemon
"(GC Worker Thread 1)" id=? idx=0x10 tid=18146 prio=5 alive, in native, daemon
"(GC Worker Thread 2)" id=? idx=0x14 tid=18147 prio=5 alive, in native, daemon
"(GC Worker Thread 3)" id=? idx=0x18 tid=18148 prio=5 alive, in native, daemon
"(GC Worker Thread 4)" id=? idx=0x1c tid=18149 prio=5 alive, in native, daemon
"(Code Generation Thread 1)" id=4 idx=0x20 tid=18150 prio=5 alive, in native, native_waiting, daemon
"(Code Optimization Thread 1)" id=5 idx=0x24 tid=18151 prio=5 alive, in native, native_waiting, daemon
"(VM Periodic Task)" id=6 idx=0x28 tid=18152 prio=10 alive, in native, daemon
"Finalizer" id=7 idx=0x2c tid=18153 prio=8 alive, in native, native_waiting, daemon
at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)
at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
at jrockit/memory/Finalizer$4.run(Finalizer.java:159)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Reference Handler" id=8 idx=0x30 tid=18154 prio=10 alive, in native, native_waiting, daemon
at java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native Method)
at java/lang/ref/Reference.access$100(Reference.java:11)
at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"(Sensor Event Thread)" id=9 idx=0x34 tid=18155 prio=5 alive, in native, daemon
"Logger@9217551 3.5/459" id=12 idx=0x38 tid=18158 prio=3 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/application/console/Coherence$Logger$Queue@0x4e1faef8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/application/console/Coherence$Logger$Queue@0x4e1faef8[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"PacketListener1" id=14 idx=0x3c tid=18159 prio=8 alive, in native, daemon
at java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native Method)
^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb21360[recursive]
at java/net/PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)[inlined]
at java/net/DatagramSocket.receive(DatagramSocket.java:712)[optimized]
^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb21360[thin lock]
^-- Holding lock: java/net/DatagramPacket@0x73a80930[thin lock]
^-- Holding lock: java/net/DatagramSocket@0x4cb21338[thin lock]
at com/tangosol/coherence/component/net/socket/UdpSocket.receive(UdpSocket.CDB:20)[optimized]
at com/tangosol/coherence/component/net/UdpPacket.receive(UdpPacket.CDB:4)[optimized]
at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketListener.onNotify(PacketListener.CDB:19)
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"PacketReceiver" id=15 idx=0x40 tid=18160 prio=7 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketReceiver$InQueue@0x4cf60d58[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketReceiver.onWait(PacketReceiver.CDB:2)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketReceiver$InQueue@0x4cf60d58[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"PacketPublisher" id=16 idx=0x44 tid=18161 prio=6 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketPublisher.onWait(PacketPublisher.CDB:2)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketPublisher$InQueue@0x4ca053a8[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"PacketSpeaker" id=17 idx=0x48 tid=18162 prio=8 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/net/Cluster$PacketSpeaker$BundlingQueue@0x4cabe9d8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/queue/ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
^-- Lock released while waiting: com/tangosol/coherence/component/net/Cluster$PacketSpeaker$BundlingQueue@0x4cabe9d8[fat lock]
at com/tangosol/coherence/component/util/queue/ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
at com/tangosol/coherence/component/util/Queue.remove(Queue.CDB:1)
at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketSpeaker.onNotify(PacketSpeaker.CDB:62)
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"PacketListenerN" id=18 idx=0x4c tid=18163 prio=8 alive, in native, daemon
at java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native Method)
^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb221b0[recursive]
at java/net/PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)[inlined]
at java/net/DatagramSocket.receive(DatagramSocket.java:712)[optimized]
^-- Holding lock: java/net/PlainDatagramSocketImpl@0x4cb221b0[thin lock]
^-- Holding lock: java/net/DatagramPacket@0x5e55a240[thin lock]
^-- Holding lock: java/net/MulticastSocket@0x4cb22178[thin lock]
at com/tangosol/coherence/component/net/socket/UdpSocket.receive(UdpSocket.CDB:20)[optimized]
at com/tangosol/coherence/component/net/UdpPacket.receive(UdpPacket.CDB:4)[optimized]
at com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketListener.onNotify(PacketListener.CDB:19)
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Cluster|Member(Id=2, Timestamp=2011-02-23 16:25:36.488, Address=10.10.100.77:8095, MachineId=25677, Location=site:emtzzzz.com,machine:njdev04,process:18142,member:cldev201~carboncachelauncher~181)" id=19 idx=0x50 tid=18164 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4cac64d8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4cac64d8[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"TcpRingListener" id=20 idx=0x54 tid=18165 prio=6 alive, in native, daemon
at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
^-- Holding lock: java/net/SocksSocketImpl@0x4cb227d8[thin lock]
at java/net/ServerSocket.implAccept(ServerSocket.java:453)
at java/net/ServerSocket.accept(ServerSocket.java:421)
at com/tangosol/coherence/component/net/socket/TcpSocketAccepter.accept(TcpSocketAccepter.CDB:18)
at com/tangosol/coherence/component/util/daemon/TcpRingListener.acceptConnection(TcpRingListener.CDB:10)
at com/tangosol/coherence/component/util/daemon/TcpRingListener.onNotify(TcpRingListener.CDB:9)
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:37)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Invocation:Management:EventDispatcher" id=22 idx=0x58 tid=18166 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher$Queue@0x4ab76140[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher$Queue@0x4ab76140[fat lock]
at com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher.onWait(Service.CDB:7)
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"Invocation:Management" id=23 idx=0x5c tid=18167 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4ab75508[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4ab75508[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
"DistributedCache" id=25 idx=0x60 tid=18168 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4a3869b0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18)[inlined]
at com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:9)[optimized]
^-- Lock released while waiting: com/tangosol/coherence/component/util/queue/concurrentQueue/DualQueue@0x4a3869b0[fat lock]
at com/tangosol/coherence/component/util/Daemon.run(Daemon.CDB:34)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
===== END OF THREAD DUMP ===============Charlie, Cameron,
Thanks for following up. To answer Cameron's question - yes, running top shows that it is this process using a significant amount of the CPU. I captured a JRA this morning setting nativesamples=true.
I see the following when I look at the Hot Methods - let me know if there is something else that would shed some more light on this issue.
Method Percent(%) #Samples Optimized Native Method Id.
libjvm.so#mmDetGcFollowReferencesWorkers 28.575 2,062 0 1 0x2AE6B82EC2C1
libpthread.so.0#__recvfrom_nocancel 26.012 1,877 0 1 0x3CC1A0D689
libpthread.so.0#__accept_nocancel 26.012 1,877 0 1 0x3CC1A0D4A9
libjvm.so#mmBalanceGetReference 4.975 359 0 1 0x2AE6B817A65B
libjvm.so#objIterStepFieldFromBitmaskWord 4.753 343 0 1 0x2AE6B82A6B34
libjvm.so#objIterStep 3.88 280 0 1 0x2AE6B82A6F8D
libjvm.so#objIterInitInner 2.744 198 0 1 0x2AE6B82A71B7
libjvm.so#mmBalanceStoreReference 1.414 102 0 1 0x2AE6B817A781
libjvm.so#objIterSetupFields 0.637 46 0 1 0x2AE6B82A6B08
libjvm.so#nativeInnerLockNotLinked 0.236 17 0 1 0x2AE6B829DE76
libjvm.so#mmWorkPacketPoolGet 0.125 9 0 1 0x2AE6B817A4F5
libjvm.so#qBitSetIsBitSetInWord 0.069 5 0 1 0x2AE6B82C8AA6
libjvm.so#nativeInnerUnlockNotLinked 0.069 5 0 1 0x2AE6B829E028
libjvm.so#mmWorkPacketPoolAdd 0.042 3 0 1 0x2AE6B817A48E
libjvm.so#objPoolMarkAllHard 0.042 3 0 1 0x2AE6B82A4923
libjvm.so#objIterSetupArray 0.028 2 0 1 0x2AE6B82A6BDB
libjvm.so#markAcquired 0.028 2 0 1 0x2AE6B829DBD3
libjvm.so#mmPointerMatrixTryInsertAtIndex 0.028 2 0 1 0x2AE6B82BEFEB
libjvm.so#mmPointerMatrixClear 0.028 2 0 1 0x2AE6B82BF61C
libjvm.so#objPoolMarkWeakConcurrently 0.028 2 0 1 0x2AE6B82A5358
libjvm.so#qBitSetFindFirstNonClearedWord 0.014 1 0 1 0x2AE6B82C888E
libjvm.so#utilCounterToNanos 0.014 1 0 1 0x2AE6B8313B53
libjvm.so#waitForLockIfNeededAndMarkAcquired 0.014 1 0 1 0x2AE6B829DC30
libjvm.so#vmtLinkData 0.014 1 0 1 0x2AE6B8325B2E
libjvm.so#mmBitsIsObjectMarkedGrey 0.014 1 0 1 0x2AE6B82ED7E9
libjvm.so#qBitSetFindLastSetBitInWord 0.014 1 0 1 0x2AE6B82C8854
libjvm.so#mark_writebarriers 0.014 1 0 1 0x2AE6B8250EEA
libjvm.so#signalNextInLockQueueIfNeeded 0.014 1 0 1 0x2AE6B829DF52
libjvm.so#mmGetUsingMatrixes 0.014 1 0 1 0x2AE6B819F33C
libjvm.so#ptGetThreadId 0.014 1 0 1 0x2AE6B82BD9ED
libc.so.6#memset 0.014 1 0 1 0x3CC0A7A000
libjvm.so#setupNodeForSelf 0.014 1 0 1 0x2AE6B829DD87
libc.so.6#_int_free 0.014 1 0 1 0x3CC0A714E0
libjvm.so#mmAddChunkToList 0.014 1 0 1 0x2AE6B830977C
libjvm.so#vmtiUnlinkData 0.014 1 0 1 0x2AE6B8325B4A
libjvm.so#nativeLockInSuspendCritical 0.014 1 0 1 0x2AE6B829E0E3
libjvm.so#mmSweepHeapPart 0.014 1 0 1 0x2AE6B830989A
libjvm.so#mmBalanceWorkSetSwapPackets 0.014 1 0 1 0x2AE6B817A647
libc.so.6#_int_malloc 0.014 1 0 1 0x3CC0A71E80
libjvm.so#charToJlcType 0.014 1 0 1 0x2AE6B8253633
unknown#unknown functions 0 0 0 1 0x2AE6B8597090
Running with nativemethods=false, I see the following under hot methods
Method Percent(%) #Samples Optimized Native Method Id.
jrockit.vm.Locks.monitorExitSecondStage(Object) 50 1 1 0 0x161D30D0
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onNotify() 50 1 1 0 0x169D2C70
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onNotify() 0 0 1 0 0x1551C050
com.tangosol.coherence.component.util.queue.ConcurrentQueue.checkFlush(int) 0 0 1 0 0x151C2180
java.lang.Thread.run() 0 0 0 0 0x14E98EB0
jrockit.vm.RNI.c2java(long, long, long, long, long) 0 0 0 0 0x14C07830
unknown#unknown functions 0 0 0 1 0x2AE6B8597090
com.tangosol.coherence.component.util.Daemon.run() 0 0 0 0 0x15CFEBC0
com.tangosol.coherence.component.util.queue.ConcurrentQueue.add(Object) 0 0 1 0 0x15ED6B00
com.tangosol.coherence.component.util.queue.ConcurrentQueue.onAddElement() 0 0 1 0 0x151C9C20
com.tangosol.coherence.component.util.queue.ConcurrentQueue.flush(boolean) 0 0 1 0 0x1618CAD0
Thanks,
Jason -
Error Manual intervention is required to stop the members...
I'm seeing a couple errors when disconnected members try to rejoin.
I've read what's here about "validate polls":
http://coherence.oracle.com/display/COH35UG/Partitioned+Cache+Service+Log+Messages
I think the relevant stuff is this:
2010-12-15 09:04:25.193/17363.862 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=n/a): validatePolls: This service timed-out due to unanswered handshake request. Manual intervention is required to stop the members that have not responded to this Poll
PollId=2, active
InitTimeMillis=1292425166002
Service=Cluster (0)
RespondedMemberSet=[1,2,3,4,5,6,7,8,9,10,11,12,17,19,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,40,41,42]
LeftMemberSet=[20,21,39]
RemainingMemberSet=[18]
2010-12-15 09:04:25.197/17363.866 Oracle Coherence GE 3.6.1.0 <Error> (thread=qReader, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=ServiceMemberSet...I have a number of these apps running simultaneously between two machines. At about the same time all the instances of the app running on one machine experienced the same issue. It looks like those members received cluster heartbeats which did not include the memeber so they left the cluster. They all then tried to rejoin but did not get a timely handshake back from a few members (20,21,39) so they just gave up. Is this correct behavior? Is there something I can do so it continues to reconnect? At a minimum can I somehow listen for this situation and do a System.exit(-1) so the apps' startup scripts know to restart them?
Thanks,
Andrew
The (almost) full log including the bit posted above is:
2010-12-15 04:15:18.478/17.147 Oracle Coherence 3.6.1.0 <Info> (thread=qReader, member=n/a): Loaded operational configuration from "jar:file:/Z:/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2010-12-15 04:15:18.498/17.167 Oracle Coherence 3.6.1.0 <Info> (thread=qReader, member=n/a): Loaded operational overrides from "jar:file:/Z:/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
Oracle Coherence Version 3.6.1.0 Build 19636
Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
2010-12-15 04:15:19.122/17.791 Oracle Coherence GE 3.6.1.0 <Info> (thread=qReader, member=n/a): Loaded cache configuration from "file:/Z:/coherence/cache-config.xml"
2010-12-15 04:15:24.853/23.522 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): Failed to satisfy the variance: allowed=16, actual=52
2010-12-15 04:15:24.853/23.522 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): Increasing allowable variance to 20
2010-12-15 04:15:25.884/24.553 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): This Member(Id=16, Timestamp=2010-12-15 04:15:29.001, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient, Edition=Grid Edition, Mode=Development, CpuCount=32, SocketCount=32) joined cluster "dev" with senior Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=8, SocketCount=2)
2010-12-15 04:15:26.214/24.883 Oracle Coherence GE 3.6.1.0 <Info> (thread=qReader, member=n/a): Started cluster Name=dev
Group{Address=225.0.0.1, Port=54321, TTL=4}
MasterMemberSet
ThisMember=Member(Id=16, Timestamp=2010-12-15 04:15:29.001, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient)
OldestMember=Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=15, BitSetCount=2
Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer)
Member(Id=2, Timestamp=2010-12-14 20:01:50.669, Address=192.168.3.20:8090, MachineId=27412, Location=machine:amd4,process:6048,member:Administrator, Role=CoherenceServer)
Member(Id=3, Timestamp=2010-12-14 20:01:51.794, Address=192.168.3.20:8092, MachineId=27412, Location=machine:amd4,process:6980,member:Administrator, Role=CoherenceServer)
Member(Id=4, Timestamp=2010-12-15 02:33:41.403, Address=192.168.3.5:8088, MachineId=27139, Location=machine:blackjack,process:12420,member:Administrator, Role=DJclientStartMe)
Member(Id=5, Timestamp=2010-12-15 02:56:49.727, Address=192.168.3.5:8090, MachineId=27139, Location=machine:blackjack,process:7520,member:Administrator, Role=DJclientBboClient)
Member(Id=7, Timestamp=2010-12-15 04:11:43.521, Address=192.168.3.5:8092, MachineId=27139, Location=machine:blackjack,process:7456,member:JD, Role=PE)
Member(Id=8, Timestamp=2010-12-15 04:15:07.121, Address=192.168.3.7:8088, MachineId=27399, Location=machine:amd2,process:8036,member:ten, Role=RediquoteclientRediQuoteClient)
Member(Id=9, Timestamp=2010-12-15 04:15:07.27, Address=192.168.3.7:8090, MachineId=27399, Location=machine:amd2,process:6328,member:twelve, Role=RediquoteclientRediQuoteClient)
Member(Id=10, Timestamp=2010-12-15 04:15:07.295, Address=192.168.3.7:8092, MachineId=27399, Location=machine:amd2,process:2028,member:three, Role=RediquoteclientRediQuoteClient)
Member(Id=11, Timestamp=2010-12-15 04:15:07.413, Address=192.168.3.7:8094, MachineId=27399, Location=machine:amd2,process:2744,member:two, Role=RediquoteclientRediQuoteClient)
Member(Id=12, Timestamp=2010-12-15 04:15:11.511, Address=192.168.3.7:8096, MachineId=27399, Location=machine:amd2,process:2704,member:one, Role=RediquoteclientRediQuoteClient)
Member(Id=13, Timestamp=2010-12-15 04:15:23.773, Address=192.168.3.6:8088, MachineId=27140, Location=machine:mothra,process:8424,member:five, Role=RediquoteclientRediQuoteClient)
Member(Id=14, Timestamp=2010-12-15 04:15:24.957, Address=192.168.3.6:8090, MachineId=27140, Location=machine:mothra,process:8480,member:four, Role=RediquoteclientRediQuoteClient)
Member(Id=15, Timestamp=2010-12-15 04:15:28.992, Address=192.168.3.6:8094, MachineId=27140, Location=machine:mothra,process:5152,member:seven, Role=RediquoteclientRediQuoteClient)
Member(Id=16, Timestamp=2010-12-15 04:15:29.001, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient)
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0,BitSetCount=0
TcpRing{Connections=[15]}
IpMonitor{AddressListSize=3}
2010-12-15 07:12:32.467/10651.136 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 86107 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 447 packets rescheduled, PauseRate=0.0189, Threshold=1878
2010-12-15 07:12:42.215/10660.884 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 49852 ms communication delay (probable remote GC) with Member(Id=20, Timestamp=2010-12-15 05:56:02.412, Address=192.168.3.20:8098, MachineId=27412, Location=machine:amd4,process:4704,member:Administrator, Role=StatsStatsLoader2); 266 packets rescheduled, PauseRate=0.0107, Threshold=1878
2010-12-15 07:16:32.750/10891.419 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 3372 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 33 packets rescheduled, PauseRate=5.0E-4, Threshold=1612
2010-12-15 07:26:31.208/11489.878 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 4819 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 41 packets rescheduled, PauseRate=0.0012, Threshold=1456
2010-12-15 07:47:21.567/12740.237 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 7115 ms communication delay (probable remote GC) with Member(Id=20, Timestamp=2010-12-15 05:56:02.412, Address=192.168.3.20:8098, MachineId=27412, Location=machine:amd4,process:4704,member:Administrator, Role=StatsStatsLoader2); 52 packets rescheduled, PauseRate=0.0085, Threshold=1612
2010-12-15 07:47:49.067/12767.736 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 8614 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 60 packets rescheduled, PauseRate=0.0142, Threshold=1696
2010-12-15 07:56:32.988/13291.657 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 11513 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 74 packets rescheduled, PauseRate=0.0147, Threshold=1612
2010-12-15 08:51:33.890/16592.559 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 33650 ms communication delay (probable remote GC) with Member(Id=20, Timestamp=2010-12-15 05:56:02.412, Address=192.168.3.20:8098, MachineId=27412, Location=machine:amd4,process:4704,member:Administrator, Role=StatsStatsLoader2); 184 packets rescheduled, PauseRate=0.0085, Threshold=1456
2010-12-15 08:52:22.802/16641.471 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 10437 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 68 packets rescheduled, PauseRate=0.0016, Threshold=1129
2010-12-15 08:53:23.232/16701.901 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 5682 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 45 packets rescheduled, PauseRate=0.0020, Threshold=1073
2010-12-15 08:53:50.802/16729.471 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 15112 ms communication delay (probable remote GC) with Member(Id=39, Timestamp=2010-12-15 07:57:53.415, Address=192.168.3.20:8102, MachineId=27412, Location=machine:amd4,process:4596,member:Administrator, Role=ex_viewer); 92 packets rescheduled, PauseRate=0.0044, Threshold=1785
2010-12-15 08:53:50.814/16729.483 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 1263 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 23 packets rescheduled, PauseRate=0.0101, Threshold=1532
2010-12-15 08:54:36.660/16775.329 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 76110 ms communication delay (probable remote GC) with Member(Id=20, Timestamp=2010-12-15 05:56:02.412, Address=192.168.3.20:8098, MachineId=27412, Location=machine:amd4,process:4704,member:Administrator, Role=StatsStatsLoader2); 396 packets rescheduled, PauseRate=0.0155, Threshold=1384
2010-12-15 08:54:36.915/16775.584 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 5302 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 43 packets rescheduled, PauseRate=0.0105, Threshold=1456
2010-12-15 08:55:28.984/16827.653 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 4293 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 38 packets rescheduled, PauseRate=0.0109, Threshold=1384
2010-12-15 08:56:07.114/16865.783 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 3385 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 33 packets rescheduled, PauseRate=0.0023, Threshold=1020
2010-12-15 08:56:55.091/16913.760 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 1398 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 23 packets rescheduled, PauseRate=0.0109, Threshold=1315
2010-12-15 08:57:36.620/16955.289 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 9830 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 66 packets rescheduled, PauseRate=0.0031, Threshold=969
2010-12-15 08:57:36.825/16955.494 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 127134 ms communication delay (probable remote GC) with Member(Id=39, Timestamp=2010-12-15 07:57:53.415, Address=192.168.3.20:8102, MachineId=27412, Location=machine:amd4,process:4596,member:Administrator, Role=ex_viewer); 650 packets rescheduled, PauseRate=0.0394, Threshold=1696
2010-12-15 08:59:22.500/17061.169 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=16): Received cluster heartbeat from the senior Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer) that does not contain this Member(Id=16, Timestamp=2010-12-15 04:15:29.001, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient); stopping cluster service.
2010-12-15 08:59:22.504/17061.173 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=16): Full Thread Dump
Thread[Thread-3,5,main]
java.lang.Object.wait(Native Method)
rediquoteclient.QuoteServerLink.pause(QuoteServerLink.java:158)
rediquoteclient.QuoteServerLink$1.run(QuoteServerLink.java:54)
java.lang.Thread.run(Thread.java:619)
Thread[Cluster|Member(Id=16, Timestamp=2010-12-15 04:15:29.001, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient),5,Cluster]
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Thread.java:1487)
com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:810)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService$SeniorMemberHeartbeat.onReceived(ClusterService.CDB:33)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:11)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onNotify(ClusterService.CDB:3)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[Finalizer,8,system]
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
2010-12-15 08:59:23.722/17062.391 Oracle Coherence GE 3.6.1.0 <Info> (thread=qReader, member=16): Restarting NamedCache: quotes.REDI
2010-12-15 08:59:23.723/17062.392 Oracle Coherence GE 3.6.1.0 <Info> (thread=qReader, member=16): Restarting Service: DistributedQuotesCacheService
2010-12-15 08:59:23.723/17062.392 Oracle Coherence GE 3.6.1.0 <Info> (thread=qReader, member=n/a): Restarting cluster
2010-12-15 08:59:25.987/17064.656 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): This Member(Id=43, Timestamp=2010-12-15 08:59:44.971, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient, Edition=Grid Edition, Mode=Development, CpuCount=32, SocketCount=32) joined cluster "dev" with senior Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=8, SocketCount=2)
2010-12-15 09:03:59.364/17338.033 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=n/a): Detected soft timeout) of {WrapperGuardable Guard{Daemon=IpMonitor} Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.6, OldestMemberId=1}}
2010-12-15 09:03:59.370/17338.039 Oracle Coherence GE 3.6.1.0 <Warning> (thread=Recovery Thread, member=n/a): Attempting recovery of Guard{Daemon=IpMonitor}
2010-12-15 09:04:25.193/17363.862 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=n/a): validatePolls: This service timed-out due to unanswered handshake request. Manual intervention is required to stop the members that have not responded to this Poll
PollId=2, active
InitTimeMillis=1292425166002
Service=Cluster (0)
RespondedMemberSet=[1,2,3,4,5,6,7,8,9,10,11,12,17,19,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,40,41,42]
LeftMemberSet=[20,21,39]
RemainingMemberSet=[18]
2010-12-15 09:04:25.197/17363.866 Oracle Coherence GE 3.6.1.0 <Error> (thread=qReader, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=ServiceMemberSet(
OldestMember=n/a
ActualMemberSet=MemberSet(Size=33, BitSetCount=2
Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer)
Member(Id=2, Timestamp=2010-12-14 20:01:50.669, Address=192.168.3.20:8090, MachineId=27412, Location=machine:amd4,process:6048,member:Administrator, Role=CoherenceServer)
Member(Id=3, Timestamp=2010-12-14 20:01:51.794, Address=192.168.3.20:8092, MachineId=27412, Location=machine:amd4,process:6980,member:Administrator, Role=CoherenceServer)
Member(Id=5, Timestamp=2010-12-15 02:56:49.727, Address=192.168.3.5:8090, MachineId=27139, Location=machine:blackjack,process:7520,member:Administrator, Role=DJclientBboClient)
Member(Id=7, Timestamp=2010-12-15 04:11:43.521, Address=192.168.3.5:8092, MachineId=27139, Location=machine:blackjack,process:7456,member:JD, Role=PE)
Member(Id=8, Timestamp=2010-12-15 04:15:07.121, Address=192.168.3.7:8088, MachineId=27399, Location=machine:amd2,process:8036,member:ten, Role=RediquoteclientRediQuoteClient)
Member(Id=9, Timestamp=2010-12-15 04:15:07.27, Address=192.168.3.7:8090, MachineId=27399, Location=machine:amd2,process:6328,member:twelve, Role=RediquoteclientRediQuoteClient)
Member(Id=10, Timestamp=2010-12-15 04:15:07.295, Address=192.168.3.7:8092, MachineId=27399, Location=machine:amd2,process:2028,member:three, Role=RediquoteclientRediQuoteClient)
Member(Id=11, Timestamp=2010-12-15 04:15:07.413, Address=192.168.3.7:8094, MachineId=27399, Location=machine:amd2,process:2744,member:two, Role=RediquoteclientRediQuoteClient)
Member(Id=12, Timestamp=2010-12-15 04:15:11.511, Address=192.168.3.7:8096, MachineId=27399, Location=machine:amd2,process:2704,member:one, Role=RediquoteclientRediQuoteClient)
Member(Id=17, Timestamp=2010-12-15 05:24:58.776, Address=192.168.3.5:8096, MachineId=27139, Location=machine:blackjack,process:9836,member:Administrator, Role=DJclientRestarter)
Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast)
Member(Id=22, Timestamp=2010-12-15 06:00:07.414, Address=192.168.3.25:8088, MachineId=27417, Location=machine:amd6,process:316,member:Administrator, Role=OmsOms)
Member(Id=23, Timestamp=2010-12-15 06:00:09.872, Address=192.168.3.7:8098, MachineId=27399, Location=machine:amd2,process:2984,member:user, Role=PEQuoteProcessor)
Member(Id=24, Timestamp=2010-12-15 06:00:51.201, Address=192.168.3.24:8088, MachineId=27416, Location=machine:Amd5,process:3096,member:St, Role=OmsOms)
Member(Id=25, Timestamp=2010-12-15 06:52:44.897, Address=192.168.3.32:8088, MachineId=27424, Location=machine:themysticx,process:3804,member:Tony, Role=OmsOms)
Member(Id=26, Timestamp=2010-12-15 07:13:05.082, Address=192.168.3.26:8088, MachineId=27418, Location=machine:dabulls,process:3256,member:St, Role=BboBBOClientMain)
Member(Id=27, Timestamp=2010-12-15 07:27:00.105, Address=192.168.3.120:8090, MachineId=27512, Location=machine:TEST-1234,process:2936,member:TEST, Role=EXV)
Member(Id=28, Timestamp=2010-12-15 07:27:00.183, Address=192.168.3.120:8088, MachineId=27512, Location=machine:TEST-1234,process:3116,member:TEST, Role=oew)
Member(Id=29, Timestamp=2010-12-15 07:27:00.251, Address=192.168.3.120:8092, MachineId=27512, Location=machine:TEST-1234,process:272,member:TEST, Role=OV)
Member(Id=30, Timestamp=2010-12-15 07:51:41.145, Address=192.168.3.116:8088, MachineId=27508, Location=machine:BE,process:2144,member:user, Role=EXV)
Member(Id=32, Timestamp=2010-12-15 07:34:03.385, Address=192.168.3.5:8098, MachineId=27139, Location=machine:blackjack,process:5836,member:WayneN, Role=PE)
Member(Id=35, Timestamp=2010-12-15 08:22:49.77, Address=192.168.3.121:8092, MachineId=27513, Location=machine:Hd,process:4052,member:Hd, Role=oew)
Member(Id=36, Timestamp=2010-12-15 07:51:41.529, Address=192.168.3.116:8090, MachineId=27508, Location=machine:BE,process:2292,member:user, Role=OV)
Member(Id=37, Timestamp=2010-12-15 07:51:41.556, Address=192.168.3.116:8091, MachineId=27508, Location=machine:BE,process:2064,member:user, Role=oew)
Member(Id=38, Timestamp=2010-12-15 07:51:41.863, Address=192.168.3.116:8094, MachineId=27508, Location=machine:BE,process:2108,member:user, Role=PE)
Member(Id=40, Timestamp=2010-12-15 08:06:00.705, Address=192.168.3.5:8100, MachineId=27139, Location=machine:blackjack,process:5484,member:ChrisL, Role=PE)
Member(Id=41, Timestamp=2010-12-15 08:22:59.722, Address=192.168.3.121:8094, MachineId=27513, Location=machine:Hd,process:2972,member:Hd, Role=PE)
Member(Id=43, Timestamp=2010-12-15 08:59:44.971, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient)
Member(Id=44, Timestamp=2010-12-15 09:01:17.056, Address=192.168.3.121:8090, MachineId=27513, Location=machine:Hd,process:860,member:Hd)
Member(Id=45, Timestamp=2010-12-15 09:01:35.962, Address=192.168.3.5:8094, MachineId=27139, Location=machine:blackjack,process:4852,member:Administrator)
Member(Id=46, Timestamp=2010-12-15 09:01:41.282, Address=192.168.3.5:8088, MachineId=27139, Location=machine:blackjack,process:12420,member:Administrator)
Member(Id=47, Timestamp=2010-12-15 09:02:44.877, Address=192.168.3.121:8088, MachineId=27513, Location=machine:Hd,process:2804,member:Hd)
MemberId/ServiceVersion/ServiceJoined/MemberState
1/3.6/Tue Dec 14 20:01:47 CST 2010/JOINED,
2/3.6/Tue Dec 14 20:01:50 CST 2010/JOINED,
3/3.6/Tue Dec 14 20:01:51 CST 2010/JOINED,
5/3.6/Wed Dec 15 02:56:49 CST 2010/JOINED,
7/3.6/Wed Dec 15 04:11:43 CST 2010/JOINED,
8/3.6/Wed Dec 15 04:15:07 CST 2010/JOINED,
9/3.6/Wed Dec 15 04:15:07 CST 2010/JOINED,
10/3.6/Wed Dec 15 04:15:07 CST 2010/JOINED,
11/3.6/Wed Dec 15 04:15:07 CST 2010/JOINED,
12/3.6/Wed Dec 15 04:15:11 CST 2010/JOINED,
17/3.6/Wed Dec 15 05:24:58 CST 2010/JOINED,
18/3.6/Wed Dec 15 05:40:07 CST 2010/JOINING,
22/3.6/Wed Dec 15 06:00:07 CST 2010/JOINED,
23/3.6/Wed Dec 15 06:00:09 CST 2010/JOINED,
24/3.6/Wed Dec 15 06:00:51 CST 2010/JOINED,
25/3.6/Wed Dec 15 06:52:44 CST 2010/JOINED,
26/3.6/Wed Dec 15 07:13:05 CST 2010/JOINED,
27/3.6/Wed Dec 15 07:27:00 CST 2010/JOINED,
28/3.6/Wed Dec 15 07:27:00 CST 2010/JOINED,
29/3.6/Wed Dec 15 07:27:00 CST 2010/JOINED,
30/3.6/Wed Dec 15 07:51:41 CST 2010/JOINED,
32/3.6/Wed Dec 15 07:34:03 CST 2010/JOINED,
35/3.6/Wed Dec 15 08:22:49 CST 2010/JOINED,
36/3.6/Wed Dec 15 07:51:41 CST 2010/JOINED,
37/3.6/Wed Dec 15 07:51:41 CST 2010/JOINED,
38/3.6/Wed Dec 15 07:51:41 CST 2010/JOINED,
40/3.6/Wed Dec 15 08:06:00 CST 2010/JOINED,
41/3.6/Wed Dec 15 08:22:59 CST 2010/JOINED,
43/3.6/Wed Dec 15 08:59:44 CST 2010/JOINED,
44/3.6/Wed Dec 15 09:01:17 CST 2010/JOINED,
45/3.6/Wed Dec 15 09:01:35 CST 2010/JOINED,
46/3.6/Wed Dec 15 09:01:41 CST 2010/JOINED,
47/3.6/Wed Dec 15 09:02:44 CST 2010/JOINED
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:636)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:7)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeService.restartService(SafeService.CDB:22)
at com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:39)
at com.tangosol.coherence.component.util.safeService.SafeCacheService.ensureRunningCacheService(SafeCacheService.CDB:3)
at com.tangosol.coherence.component.util.SafeNamedCache$CacheAction.run(SafeNamedCache.CDB:3)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:337)
at com.tangosol.coherence.component.util.SafeNamedCache.restartNamedCache(SafeNamedCache.CDB:8)
at com.tangosol.coherence.component.util.SafeNamedCache.ensureRunningNamedCache(SafeNamedCache.CDB:33)
at com.tangosol.coherence.component.util.SafeNamedCache.getRunningNamedCache(SafeNamedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.putAll(SafeNamedCache.CDB:1)
at rediquoteclient.CoherencePublisher$1.run(CoherencePublisher.java:105)
at java.lang.Thread.run(Thread.java:619)Ashbaernon said:Â Firstly, the complete error message is "This VI
is not executable. The full development version of Labview is required
to fix the errors"
"VI is not executable" can mean the VI is missing some component, but it can also mean the VI or one of its subVIs has not been compiled to the same version as your RTE (Run-Time Engine). I've specifically seen this using the TestStand full-featured LabVIEW user interface, which was last compiled with LabVIEW 7.x (5 years ago), yet shipped with TS 4. Anyhow, if this is the cause, the solution is to:Â
Open the top level VI and any dynamically called VIs (i.e. VIs called by VI Server).
Holding down Ctrl and Shift, press the Run arrow to recompile all VIs in memory.
File >> Save All (or Ctrl-Shift-S)
Rebuild your EXE.
-Jason
Message Edited by LabBEAN on 02-14-2009 10:28 PM
Certified LabVIEW Architect
Wait for Flag / Set Flag
Separate Views from Implementation for Strict Type Defs -
How to properly implement custom ServiceFailurePolicy
I'd like to use Coherence service guardian to interrupt & cancel hung cacheloader threads, which are going into infinite loop due to database connection became unresponsive.
For that purpose I've tried first to create working testcase in order to learn how service guardian works.
I've created a basic distributed cache with fake cache loader to imitate a delay in response, then an infinite loop, and custom test service guardian policy.
I do see cache loader being interrupted by a call to onGuardableRecovery, but then strange thing happens: despite cacheloader thread being successfully released, service guardian still thinks that the service is stuck and tries several times to recover by calling onGuardableRecovery and finally to terminate service with the call to onGuardableTerminate.
What I'm missing here or doing wrong ?
Below are testcase classes & cache config:
package test.guardian;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.cache.CacheStore;
import com.tangosol.util.Base;
import java.util.Collection;
import java.util.Map;
public class testCacheStore extends Base implements CacheStore {
private String datasource_name = null;
public static volatile boolean loop_thread = true;
public testCacheStore(String p_datasource_name) {
CacheFactory.log("inside cacheloader constructor, p_datasource_name="+p_datasource_name,CacheFactory.LOG_DEBUG);
this.datasource_name = p_datasource_name;
public Object load(Object oKey) {
CacheFactory.log("inside cacheloader",CacheFactory.LOG_DEBUG);
try {
Thread.sleep(120*1000);
catch (InterruptedException e) {
CacheFactory.log("loader thread was interrupted"+"\n"+CacheFactory.getStackTrace(e),CacheFactory.LOG_DEBUG);
while (testCacheStore.loop_thread) {String n = "1"+"2";}
CacheFactory.log("loader stop_thread flag was set to stop thread",CacheFactory.LOG_DEBUG);
CacheFactory.log("return value from the store",CacheFactory.LOG_DEBUG);
return ((String) oKey)+"-value";
public Map loadAll(Collection colKeys) {
throw new UnsupportedOperationException();
public void store(Object oKey, Object oValue) {
public void storeAll(Map mapEntries) {
throw new UnsupportedOperationException();
public void erase(Object oKey) {
public void eraseAll(Collection colKeys) {
CacheFactory.log("inside eraseAll method", CacheFactory.LOG_DEBUG);
throw new UnsupportedOperationException();
package test.guardian;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.Guardable;
import com.tangosol.net.ServiceFailurePolicy;
import com.tangosol.net.Service;
import com.tangosol.net.Cluster;
public class mycustomGPolicy implements ServiceFailurePolicy {
public static mycustomGPolicy getInstance() {
return new mycustomGPolicy();
public mycustomGPolicy(String p_info) {
CacheFactory.log("parametrized policy class instance created",CacheFactory.LOG_DEBUG);
CacheFactory.log("p_info="+p_info,CacheFactory.LOG_DEBUG);
public mycustomGPolicy() {
CacheFactory.log("policy class instance created",CacheFactory.LOG_DEBUG);
//CacheFactory.log("p_info="+p_info,CacheFactory.LOG_DEBUG);
public void onGuardableRecovery(Guardable p_guardable, Service p_service) {
CacheFactory.log("inside onGuardableRecovery",CacheFactory.LOG_DEBUG);
CacheFactory.log("p_guardable="+p_guardable.toString(),CacheFactory.LOG_DEBUG);
CacheFactory.log("p_guardable.getClass().getCanonicalName()="+p_guardable.getClass().getCanonicalName(),CacheFactory.LOG_DEBUG);
testCacheStore.loop_thread = false;
p_guardable.recover();
p_guardable.getContext().heartbeat();
public void onGuardableTerminate(Guardable p_guardable, Service p_service) {
CacheFactory.log("inside onGuardableTerminate",CacheFactory.LOG_DEBUG);
CacheFactory.log("p_guardable="+p_guardable.toString(),CacheFactory.LOG_DEBUG);
CacheFactory.log("p_guardable.getClass().getCanonicalName()="+p_guardable.getClass().getCanonicalName(),CacheFactory.LOG_DEBUG);
testCacheStore.loop_thread = false;
p_guardable.terminate();
public void onServiceFailed(Cluster p_cluster) {
CacheFactory.log("inside onServiceFailed",CacheFactory.LOG_DEBUG);
package test.guardian;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.NamedCache;
public class testGuardianCache {
public static void main(String[] args) {
try {
log("building cache instance");
NamedCache cache = CacheFactory.getCache("test-guardable");
log("requesting cache data");
String resp4 = (String)cache.get("test1223g");
log("response:");
System.out.println(resp4+"\n");
} catch (com.tangosol.io.pof.PortableException e) {
Throwable e_src = e;
while (e_src.getCause() != null ) {
e_src = e_src.getCause();
e_src.printStackTrace();
public static void log(String info) {
System.out.println(info);
<?xml version="1.0" encoding="UTF-8"?>
<cache-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>test-guardable</cache-name>
<scheme-name>test-guardable-schema</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>test-guardable-schema</scheme-name>
<service-name>test-guardable-service</service-name>
<backing-map-scheme>
<read-write-backing-map-scheme>
<cachestore-timeout>0</cachestore-timeout>
<internal-cache-scheme>
<external-scheme>
<scheme-name>LocalNioLtd</scheme-name>
<high-units>10000</high-units>
<unit-calculator>FIXED</unit-calculator>
<nio-memory-manager>
<initial-size>10M</initial-size>
<maximum-size>50M</maximum-size>
</nio-memory-manager>
</external-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>test.guardian.testCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>jdbc-coherence/MY-CACHE-DS</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<rollback-cachestore-failures>true</rollback-cachestore-failures>
<read-only>true</read-only>
<write-delay-seconds>0</write-delay-seconds>
</read-write-backing-map-scheme>
</backing-map-scheme>
<listener/>
<autostart>true</autostart>
<thread-count>10</thread-count>
<guardian-timeout>10000</guardian-timeout>
<service-failure-policy>
<class-name>test.guardian.mycustomGPolicy</class-name>
<init-params>
<init-param>
<param-name>datasource</param-name>
<param-type>java.lang.String</param-type>
<param-value>jdbc-coherence/MY-CACHE-DS</param-value>
</init-param>
</init-params>
</service-failure-policy>
</distributed-scheme>
</caching-schemes>
</cache-config>
<?xml version="1.0"?>
<coherence>
<services>
<service>
<service-type>DistributedCache</service-type>
<service-component>DistributedCache</service-component>
<init-params>
<init-param id="4">
<param-name>local-storage</param-name>
<param-value system-property="tangosol.coherence.distributed.localstorage">false</param-value>
</init-param>
</init-params>
</service>
</services>
<cluster-config>
<member-identity>
<cluster-name>WORKSTATION</cluster-name>
</member-identity>
<unicast-listener>
<address system-property="tangosol.coherence.localhost">localhost</address>
<port system-property="tangosol.coherence.localport">7070</port>
<port-auto-adjust system-property="tangosol.coherence.localport.adjust">true</port-auto-adjust>
<well-known-addresses>
<socket-address id="1"><address>localhost</address><port>7070</port></socket-address>
</well-known-addresses>
</unicast-listener>
<authorized-hosts/>
<logging-config>
<severity-level system-property="tangosol.coherence.log.level">5</severity-level>
<character-limit system-property="tangosol.coherence.log.limit">0</character-limit>
</logging-config>
</cluster-config>
<logging-config>
<severity-level>5</severity-level>
<message-format>{date} <{level}> (thread={thread}, member={member}): {text}</message-format>
<character-limit>8192</character-limit>
</logging-config>
</coherence>
setlocal
set COHERENCE_HOME=c:\coherence36
set PROG_HOME=C:\coherence36\deploy
set COH_OPTS=-cp %COHERENCE_HOME%\lib\coherence.jar;%PROG_HOME%\testguardable.jar
set JMX_OPTS=-Dcom.sun.management.jmxremote -Dtangosol.coherence.management=all -Dtangosol.coherence.management.remote=true -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false
set OVERRIDE_OPTS=-Dtangosol.coherence.override=%PROG_HOME%\tangosol-coherence-override.xml
rem
rem
start "1testguardable" /MAX C:\jdev10g\jdk\jre\bin\java.exe %COH_OPTS% %OVERRIDE_OPTS% -Xms256m -Xmx256m -Xloggc: -Dtangosol.coherence.distributed.localstorage=true -Dtangosol.coherence.cacheconfig=%PROG_HOME%\testguardable-cache-config.xml com.tangosol.net.DefaultCacheServerINFO - 2010-07-23 23:43:07.906 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/C:/coherence36/lib/coherence.jar!/tangosol-coherence.xml"
INFO - 2010-07-23 23:43:07.921 <Info> (thread=main, member=n/a): Loaded operational overrides from "file:/C:/-PRG/coherence_ws/Guardian/deploy/tangosol-coherence-override.xml"
DEBUG - 2010-07-23 23:43:07.937 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
DEBUG -
Oracle Coherence Version 3.6.0.0 Build 17229
Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
INFO - 2010-07-23 23:43:08.531 <Info> (thread=main, member=n/a): Loaded cache configuration from "file:/C:/-PRG/coherence_ws/Guardian/deploy/testguardable-cache-config.xml"
DEBUG - 2010-07-23 23:43:09.515 <D4> (thread=main, member=n/a): TCMP bound to /192.168.1.10:7070 using SystemSocketProvider
INFO - 2010-07-23 23:43:41.000 <Info> (thread=Cluster, member=n/a): Created a new cluster "WORKSTATION" with Member(Id=1, Timestamp=2010-07-23 23:43:09.531, Address=192.168.1.10:7070, MachineId=26890, Location=site:local,machine:WS,process:5880, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=1) UID=0xC0A8010A0000012A01427FDB690A1B9E
INFO - 2010-07-23 23:43:41.015 <Info> (thread=main, member=n/a): Started cluster Name=WORKSTATION
WellKnownAddressList(Size=1,
WKA{Address=192.168.1.10, Port=7070}
MasterMemberSet
ThisMember=Member(Id=1, Timestamp=2010-07-23 23:43:09.531, Address=192.168.1.10:7070, MachineId=26890, Location=site:local,machine:WS,process:5880, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2010-07-23 23:43:09.531, Address=192.168.1.10:7070, MachineId=26890, Location=site:local,machine:WS,process:5880, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2010-07-23 23:43:09.531, Address=192.168.1.10:7070, MachineId=26890, Location=site:local,machine:WS,process:5880, Role=CoherenceServer)
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0, BitSetCount=0
TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}
DEBUG - 2010-07-23 23:43:41.093 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
DEBUG - 2010-07-23 23:43:41.718 <D5> (thread=main, member=1): parametrized policy class instance created
DEBUG - 2010-07-23 23:43:41.718 <D5> (thread=main, member=1): p_info=jdbc-coherence/MY-CACHE-DS
DEBUG - 2010-07-23 23:43:41.750 <D5> (thread=DistributedCache:test-guardable-service, member=1): Service test-guardable-service joined the cluster with senior service member 1
INFO - 2010-07-23 23:43:41.890 <Info> (thread=main, member=1):
Services
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.6, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
PartitionedCache{Name=test-guardable-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
Started DefaultCacheServer...
DEBUG - 2010-07-23 23:44:50.250 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2010-07-23 23:44:50.258, Address=192.168.1.10:7072, MachineId=26890, Location=site:local,machine:WS,process:3376, Role=TestGuardianTestGuardianCache) joined Cluster with senior member 1
DEBUG - 2010-07-23 23:44:50.390 <D5> (thread=Cluster, member=1): Member 2 joined Service Management with senior member 1
DEBUG - 2010-07-23 23:44:51.093 <D5> (thread=Cluster, member=1): Member 2 joined Service test-guardable-service with senior member 1
DEBUG - 2010-07-23 23:44:51.218 <D5> (thread=DistributedCache:test-guardable-service, member=1): inside cacheloader constructor, p_datasource_name=jdbc-coherence/MY-CACHE-DS
DEBUG - 2010-07-23 23:44:51.359 <D5> (thread=test-guardable-serviceWorker:4, member=1): inside cacheloader
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=Recovery Thread, member=1): inside onGuardableRecovery
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=Recovery Thread, member=1): p_guardable=Guard{Daemon=test-guardable-serviceWorker:4}
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=Recovery Thread, member=1): p_guardable.getClass().getCanonicalName()=com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Daemon$Guard
ERROR - 2010-07-23 23:45:00.375 <Error> (thread=DistributedCache:test-guardable-service, member=1): Attempting recovery (due to soft timeout) of {WrapperGuardable Guard{Daemon=test-guardable-serviceWorker:4} Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}}
WARN - 2010-07-23 23:45:00.375 <Warning> (thread=Recovery Thread, member=1): A worker thread has been executing task: Message "GetRequest"
FromMember=Member(Id=2, Timestamp=2010-07-23 23:44:50.258, Address=192.168.1.10:7072, MachineId=26890, Location=site:local,machine:WS,process:3376, Role=TestGuardianTestGuardianCache)
FromMessageId=29
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=59
ToPollId=0
Poll=null
Packets
[000]=Directed{PacketType=0x0DDF00D5, ToId=1, FromId=2, Direction=Incoming, ReceivedMillis=23:44:51.343, ToMemberSet=null, ServiceId=2, MessageType=59, FromMessageId=29, ToMessageId=27, MessagePartCount=1, MessagePartIndex=0, NackInProgress=false, ResendScheduled=none, Timeout=none, PendingResendSkips=0, DeliveryState=unsent, Body=0}
Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
ToMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2010-07-23 23:43:09.531, Address=192.168.1.10:7070, MachineId=26890, Location=site:local,machine:WS,process:5880, Role=CoherenceServer)
NotifySent=false
} for 9016ms and appears to be stuck; attempting to interrupt: test-guardable-serviceWorker:4
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=test-guardable-serviceWorker:4, member=1): loader thread was interrupted
at java.lang.Thread.sleep(Native Method)
at test.guardian.testCacheStore.load(testCacheStore.java:26)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.load(ReadWriteBackingMap.java:4759)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.loadInternal(ReadWriteBackingMap.java:4344)
at com.tangosol.net.cache.ReadWriteBackingMap.get(ReadWriteBackingMap.java:807)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onGetRequest(PartitionedCache.CDB:22)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$GetRequest.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:595)
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=test-guardable-serviceWorker:4, member=1): loader stop_thread flag was set to stop thread
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=test-guardable-serviceWorker:4, member=1): return value from the store
DEBUG - 2010-07-23 23:45:00.390 <D5> (thread=Cluster, member=1): TcpRing disconnected from Member(Id=2, Timestamp=2010-07-23 23:44:50.258, Address=192.168.1.10:7072, MachineId=26890, Location=site:local,machine:WS,process:3376, Role=TestGuardianTestGuardianCache) due to a peer departure; removing the member.
DEBUG - 2010-07-23 23:45:00.390 <D5> (thread=Cluster, member=1): Member 2 left service Management with senior member 1
DEBUG - 2010-07-23 23:45:00.390 <D5> (thread=Cluster, member=1): Member 2 left service test-guardable-service with senior member 1
DEBUG - 2010-07-23 23:45:00.390 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2010-07-23 23:45:00.39, Address=192.168.1.10:7072, MachineId=26890, Location=site:local,machine:WS,process:3376, Role=TestGuardianTestGuardianCache) left Cluster with senior member 1
ERROR - 2010-07-23 23:48:16.250 <Error> (thread=Cluster, member=1): Attempting recovery (due to soft timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:test-guardable-service} Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}}
DEBUG - 2010-07-23 23:48:16.250 <D5> (thread=Recovery Thread, member=1): inside onGuardableRecovery
DEBUG - 2010-07-23 23:48:16.250 <D5> (thread=Recovery Thread, member=1): p_guardable=Guard{Daemon=DistributedCache:test-guardable-service}
DEBUG - 2010-07-23 23:48:16.250 <D5> (thread=Recovery Thread, member=1): p_guardable.getClass().getCanonicalName()=com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard
DEBUG - Interrupted PartitionedCache, Thread[DistributedCache:test-guardable-service,5,Cluster]
ERROR - 2010-07-23 23:48:46.765 <Error> (thread=Cluster, member=1): Terminating guarded execution (due to hard timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:test-guardable-service} Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}}
DEBUG - 2010-07-23 23:48:46.765 <D5> (thread=Termination Thread, member=1): inside onGuardableTerminate
DEBUG - 2010-07-23 23:48:46.765 <D5> (thread=Termination Thread, member=1): p_guardable=Guard{Daemon=DistributedCache:test-guardable-service}
DEBUG - 2010-07-23 23:48:46.765 <D5> (thread=Termination Thread, member=1): p_guardable.getClass().getCanonicalName()=com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard
DEBUG - 2010-07-23 23:48:46.765 <D5> (thread=DistributedCache:test-guardable-service, member=1): Service test-guardable-service left the cluster
INFO - 2010-07-23 23:48:46.890 <Info> (thread=main, member=1): Restarting Service: test-guardable-service
DEBUG - 2010-07-23 23:48:46.906 <D5> (thread=main, member=1): parametrized policy class instance created
DEBUG - 2010-07-23 23:48:46.906 <D5> (thread=main, member=1): p_info=jdbc-coherence/MY-CACHE-DS
DEBUG - 2010-07-23 23:48:46.906 <D5> (thread=DistributedCache:test-guardable-service, member=1): Service test-guardable-service joined the cluster with senior service member 1
ERROR - 2010-07-23 23:48:55.265 <Error> (thread=Cluster, member=1): Attempting recovery (due to soft timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:test-guardable-service} Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STOPPED), Not initialized}}
DEBUG - 2010-07-23 23:48:55.265 <D5> (thread=Recovery Thread, member=1): inside onGuardableRecovery
DEBUG - 2010-07-23 23:48:55.265 <D5> (thread=Recovery Thread, member=1): p_guardable=Guard{Daemon=DistributedCache:test-guardable-service}
DEBUG - 2010-07-23 23:48:55.265 <D5> (thread=Recovery Thread, member=1): p_guardable.getClass().getCanonicalName()=com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard
ERROR - 2010-07-23 23:48:56.281 <Error> (thread=Cluster, member=1): Terminating guarded execution (due to hard timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:test-guardable-service} Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STOPPED), Not initialized}}
DEBUG - 2010-07-23 23:48:56.281 <D5> (thread=Termination Thread, member=1): inside onGuardableTerminate
DEBUG - 2010-07-23 23:48:56.281 <D5> (thread=Termination Thread, member=1): p_guardable=Guard{Daemon=DistributedCache:test-guardable-service}
DEBUG - 2010-07-23 23:48:56.281 <D5> (thread=Termination Thread, member=1): p_guardable.getClass().getCanonicalName()=com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard -
Stuck Threads Leading to server hang
Hi ,
We have been facing a problem with Stuck Threads in our production environment. We are currently running version 8.1. Though the stuck Threads do point to JDBC issues, we suspect that these problems may be related to some Weblogic/OS configuration issue as well. These probelms eventually lead to a server hang.
The general pattern of the problem is that we see messages like :
Dec 24, 2006 8:56:09 AM EST> <Debug> <Common> <BEA-000610> < RP(rtqdbConnectionPool):timeoutInactiveRes (20)>
####<Dec 25, 2006 8:54:51 AM EST> <Error> <WebLogicServer> <hygeia> <hygeia-rtq-vm7> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <UNKNOWN_SESSION has been busy for "896" seconds working on the request "Http Request: /quote.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ==================== This email/fax message is for the sole use of the intended
Sometimes a deadlock Exception would be thrown from the Database,
nested exception is java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource
But this is not as often as we get the Stuck Thread messages.
This is a typical Thread dump from one of our servers when it complained of a stuck thread.
///*************Thread Dump Start**********************///
Full thread dump Java HotSpot(TM) Server VM (1.4.2_08-b03 mixed mode):
"idle thread" daemon prio=5 tid=0x00d2c150 nid=0x7f in Object.wait() [94e7f000..94e7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb2010000> (a org.xbill.Task.WorkerThread)
at org.xbill.Task.WorkerThread.run(WorkerThread.java:130)
- locked <0xb2010000> (a org.xbill.Task.WorkerThread)
"Thread-28" daemon prio=5 tid=0x00c6a7c0 nid=0x7d in Object.wait() [92c7f000..92c7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xbce667f8> (a com.geico.util.LockManager)
at com.geico.util.LockManager.waitForWork(LockManager.java:85)
- locked <0xbce667f8> (a com.geico.util.LockManager)
at com.geico.mail.Mail.run(Mail.java:259)
at java.lang.Thread.run(Thread.java:534)
"CacheManager-idle" daemon prio=5 tid=0x00c7d950 nid=0x7c in Object.wait() [92d7f000..92d7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xbce66878> (a com.geico.util.LockManager)
at com.geico.util.LockManager.waitForWork(LockManager.java:85)
- locked <0xbce66878> (a com.geico.util.LockManager)
at com.geico.cache.CacheManager.run(CacheManager.java:353)
at java.lang.Thread.run(Thread.java:534)
"ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x006a6c90 nid=0x7a in Object.wait() [92e7f000..92e7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5db8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x0120c568 nid=0x79 in Object.wait() [92f7f000..92f7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5e38> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=5 tid=0x006a6af0 nid=0x78 in Object.wait() [9307f000..9307fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5eb8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'JMSStore<AptSoftFileStore-morpheus-rtq-vm7>.ioThreadPool'" daemon prio=5 tid=0x0052a198 nid=0x77 in Object.wait() [9317f000..9317fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xbc2a5f38> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"Thread-26" daemon prio=5 tid=0x00635d58 nid=0x75 in Object.wait() [9327f000..9327fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xbb97ad20> (a com.geico.util.LockManager)
at java.lang.Object.wait(Object.java:429)
at com.geico.util.LockManager.waitForWork(LockManager.java:63)
- locked <0xbb97ad20> (a com.geico.util.LockManager)
at com.geico.util.providers.StorageProvider.run(StorageProvider.java:158)
at java.lang.Thread.run(Thread.java:534)
"Thread-25" daemon prio=5 tid=0x00fac430 nid=0x72 waiting on condition [9337f000..9337fc28]
at java.lang.Thread.sleep(Native Method)
at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1080)
at java.lang.Thread.run(Thread.java:534)
"ListenThread.Default" prio=5 tid=0x014ec6e8 nid=0x71 runnable [9347f000..9347fc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xbb97ae60> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
"Thread-24" daemon prio=5 tid=0x011e9668 nid=0x70 waiting on condition [9357f000..9357fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.HeartbeatLogger$1.run(HeartbeatLogger.java:133)
"Thread-22" daemon prio=5 tid=0x011e8610 nid=0x6f waiting on condition [9367f000..9367fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.StreamLogger.pause(StreamLogger.java:263)
at com.geico.log.StreamLogger.access$300(StreamLogger.java:15)
at com.geico.log.StreamLogger$WorkerThread.run(StreamLogger.java:217)
at java.lang.Thread.run(Thread.java:534)
"filewatcher" daemon prio=5 tid=0x0111ca50 nid=0x6e waiting on condition [9377f000..9377fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.watcher.FileWatcher.sleep(FileWatcher.java:135)
at com.geico.watcher.FileWatcher.run(FileWatcher.java:118)
at java.lang.Thread.run(Thread.java:534)
"Thread-18" prio=5 tid=0x00de2388 nid=0x6d in Object.wait() [9437e000..9437fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xa20f1a60> (a weblogic.jms.backend.BEConsumerReceiveRequest)
at java.lang.Object.wait(Object.java:429)
at weblogic.jms.dispatcher.Request.sleepTillNotified(Request.java:240)
at weblogic.jms.dispatcher.Request.wrappedFiniteStateMachine(Request.java:685)
- locked <0xa20f1a60> (a weblogic.jms.backend.BEConsumerReceiveRequest)
at weblogic.jms.dispatcher.DispatcherImpl.dispatchAsyncInternal(DispatcherImpl.java:132)
at weblogic.jms.dispatcher.DispatcherImpl.dispatchAsync(DispatcherImpl.java:123)
at weblogic.jms.dispatcher.Request.dispatchAsync(Request.java:878)
at weblogic.jms.frontend.FEConsumer.receive(FEConsumer.java:452)
at weblogic.jms.frontend.FEConsumer.invoke(FEConsumer.java:643)
at weblogic.jms.dispatcher.Request.wrappedFiniteStateMachine(Request.java:643)
at weblogic.jms.dispatcher.DispatcherImpl.dispatchSyncTran(DispatcherImpl.java:192)
at weblogic.jms.client.JMSSession.receiveMessage(JMSSession.java:456)
at weblogic.jms.client.JMSConsumer.receive(JMSConsumer.java:421)
at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:800)
at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:783)
at org.springframework.jms.core.JmsTemplate$8.doInJms(JmsTemplate.java:741)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:582)
at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:739)
at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:731)
at com.geico.uierrorlog.UiErrorUtil.getMessage(UiErrorUtil.java:89)
at com.geico.uierrorlog.UiErrorDaemon.getMessage(UiErrorDaemon.java:108)
at com.geico.uierrorlog.UiErrorDaemon.run(UiErrorDaemon.java:166)
at java.lang.Thread.run(Thread.java:534)
"StatisticsThread" daemon prio=5 tid=0x00d0c798 nid=0x6c in Object.wait() [9447f000..9447fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7b18> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee7b18> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"StatisticsThread" daemon prio=5 tid=0x01202c18 nid=0x6b in Object.wait() [9457f000..9457fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7bd8> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee7bd8> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"StatisticsThread" daemon prio=5 tid=0x00b33f18 nid=0x6a in Object.wait() [9467f000..9467fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7c90> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee7c90> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"DistributedCache:DistributedRTQOverflow" daemon prio=5 tid=0x009895e8 nid=0x69 in Object.wait() [9477f000..9477fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7d40> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee7d40> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:PoolStatsDistributedCache:EventDispatcher" daemon prio=5 tid=0x00dff060 nid=0x68 in Object.wait() [9487f000..9487fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee7f18> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee7f18> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:PoolStatsDistributedCache" daemon prio=5 tid=0x006f3e98 nid=0x67 in Object.wait() [9497f000..9497fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee7fe8> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"StatisticsThread" daemon prio=5 tid=0x0090d8d0 nid=0x66 in Object.wait() [94a7f000..94a7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee81c8> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee81c8> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"DistributedCache:dist-RTQSession:EventDispatcher" daemon prio=5 tid=0x00a05110 nid=0x65 in Object.wait() [94b7f000..94b7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8278> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8278> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-RTQSession" daemon prio=5 tid=0x0036e600 nid=0x64 in Object.wait() [94c7f000..94c7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8348> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"ReplicatedCache" daemon prio=5 tid=0x00e9f9f8 nid=0x63 in Object.wait() [94d7f000..94d7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8520> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Scheduler_Worker-9" prio=5 tid=0x00339820 nid=0x61 in Object.wait() [94f7f000..94f7fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-8" prio=5 tid=0x0033af58 nid=0x60 in Object.wait() [9507f000..9507fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-7" prio=5 tid=0x0033a878 nid=0x5f in Object.wait() [9517f000..9517fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-6" prio=5 tid=0x00935488 nid=0x5e in Object.wait() [9527f000..9527fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-5" prio=5 tid=0x00bb2190 nid=0x5d in Object.wait() [9537f000..9537fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-4" prio=5 tid=0x010b9038 nid=0x5c in Object.wait() [9547f000..9547fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-3" prio=5 tid=0x00a34678 nid=0x5b in Object.wait() [9557f000..9557fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-2" prio=5 tid=0x014f9860 nid=0x5a in Object.wait() [9567f000..9567fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-1" prio=5 tid=0x0044e530 nid=0x59 in Object.wait() [9577f000..9577fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"Scheduler_Worker-0" prio=5 tid=0x0110b8e8 nid=0x58 in Object.wait() [9587f000..9587fc28]
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:423)
- locked <0xb8ee8788> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:53)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:514)
"StatisticsThread" daemon prio=5 tid=0x011352b8 nid=0x57 in Object.wait() [9597f000..9597fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8b20> (a com.geico.tangosol.stats.StatsDaemon)
at com.geico.tangosol.stats.StatsDaemon.run(StatsDaemon.java:144)
- locked <0xb8ee8b20> (a com.geico.tangosol.stats.StatsDaemon)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
"DistributedCache:dist-rtq-sessionheader:EventDispatcher" daemon prio=5 tid=0x004222a0 nid=0x56 in Object.wait() [95a7f000..95a7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8bd8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8bd8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-rtq-sessionheader" daemon prio=5 tid=0x01278fa0 nid=0x55 in Object.wait() [95b7f000..95b7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8ca8> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-hibernate-default:EventDispatcher" daemon prio=5 tid=0x015273f0 nid=0x54 in Object.wait() [95c7f000..95c7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8ee8e80> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8e80> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"DistributedCache:dist-hibernate-default" daemon prio=5 tid=0x00ea1328 nid=0x53 in Object.wait() [9607f000..9607fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8ee8f50> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Invocation:Management" daemon prio=5 tid=0x003d8968 nid=0x52 in Object.wait() [95d7f000..95d7fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448698> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Invocation:Management:EventDispatcher" daemon prio=5 tid=0x007e6040 nid=0x51 in Object.wait() [95e7f000..95e7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb84487f0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb84487f0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"TcpRingListener" daemon prio=6 tid=0x007e6f40 nid=0x50 runnable [95f7f000..95f7fc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xb84488d0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at com.tangosol.coherence.component.net.socket.TcpSocketAccepter.accept(TcpSocketAccepter.CDB:17)
at com.tangosol.coherence.component.util.daemon.TcpRingListener.acceptConnection(TcpRingListener.CDB:9)
at com.tangosol.coherence.component.util.daemon.TcpRingListener.onNotify(TcpRingListener.CDB:1)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:34)
at java.lang.Thread.run(Thread.java:534)
"Cluster" daemon prio=5 tid=0x00c94738 nid=0x4e in Object.wait() [9617f000..9617fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448ac0> (a com.tangosol.coherence.component.net.Cluster$ClusterService$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketPublisher" daemon prio=6 tid=0x00d83418 nid=0x4d runnable [9627f000..9627fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448c78> (a com.tangosol.coherence.component.net.Cluster$PacketPublisher$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketReceiver" daemon prio=7 tid=0x00b3cd70 nid=0x4c in Object.wait() [9637f000..9637fc28]
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8448dc0> (a com.tangosol.coherence.component.net.Cluster$PacketReceiver$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketListenerN" daemon prio=8 tid=0x003d7cd0 nid=0x4b runnable [9647f000..9647fc28]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xb84494c8> (a java.net.DatagramPacket)
- locked <0xb84494e8> (a java.net.MulticastSocket)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:18)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:17)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onWait(PacketListener.CDB:5)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"PacketListener1" daemon prio=8 tid=0x00df5538 nid=0x4a runnable [9657f000..9657fc28]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xb8449cb8> (a java.net.DatagramPacket)
- locked <0xb8449cd8> (a java.net.DatagramSocket)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:18)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:17)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onWait(PacketListener.CDB:5)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"Logger@13785534 3.1.2/343" daemon prio=3 tid=0x006b7358 nid=0x49 in Object.wait() [9667f000..9667fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8449e90> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:9)
- locked <0xb8449e90> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:31)
at java.lang.Thread.run(Thread.java:534)
"ExecuteThread: '5' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x011384b0 nid=0x48 in Object.wait() [9677f000..9677fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c91f8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '4' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x01137bc8 nid=0x47 in Object.wait() [9687f000..9687fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c9278> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x0111d4e0 nid=0x46 in Object.wait() [9697f000..9697fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c92f8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x010dd218 nid=0x45 in Object.wait() [96a7f000..96a7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c9378> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00bb1c98 nid=0x44 in Object.wait() [96b7f000..96b7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c93f8> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00979ac8 nid=0x43 in Object.wait() [96c7f000..96c7fc28]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0xb82c9478> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"Thread-12" daemon prio=5 tid=0x00200610 nid=0x42 waiting on condition [96d7f000..96d7fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.HeartbeatLogger$1.run(HeartbeatLogger.java:133)
"Thread-10" daemon prio=5 tid=0x00200470 nid=0x41 waiting on condition [96e7f000..96e7fc28]
at java.lang.Thread.sleep(Native Method)
at com.geico.log.StreamLogger.pause(StreamLogger.java:267)
at com.geico.log.StreamLogger.access$300(StreamLogger.java:19)
at com.geico.log.StreamLogger$WorkerThread.run(StreamLogger.java:221)
at java.lang.Thread.run(Thread.java:534)
"ClockDaemonThread-0" daemon prio=5 tid=0x00ab6a48 nid=0x40 in Object.wait() [96f7f000..96f7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8051b20> (a com.geico.util.concurrent.ClockDaemon)
at com.geico.util.concurrent.ClockDaemon.nextTask(ClockDaemon.java:328)
- locked <0xb8051b20> (a com.geico.util.concurrent.ClockDaemon)
at com.geico.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:362)
at java.lang.Thread.run(Thread.java:534)
"Thread-9" prio=5 tid=0x008912e0 nid=0x3f in Object.wait() [9ad7f000..9ad7fc28]
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xb8051bc0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=5 tid=0x001d7fa8 nid=0x3e in Object.wait() [9ae7f000..9ae7fc28]
at java.lang.Object.wait(Native Method)
at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
- locked <0xb8051c90> (a weblogic.jms.backend.BETimerTree)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
"Thread-8" prio=5 tid=0x00247418 nid=0x3d in Object.wait() [9af7f000..9af7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0xb8051d20> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xb8051d20> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"Thread-7" prio=5 tid=0x001d8a28 nid=0x3c in Object.wait() [9b07f000..9b07fc28]
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xb8051dc0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"ExecuteThread: '0' for queue: 'weblogic.cluster.MulticastManager'" daemon prio=5 tid=0x00da0a60 nid=0x3b runnable [9b17f000..9b17fc28]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xa22dc3e8> (a java.net.DatagramPacket)
- locked <0xb7558130> (a java.net.MulticastSocket)
at weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:169)
at weblogic.cluster.MulticastManager.execute(MulticastManager.java:395)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
"SSLListenThread.Administrator" prio=5 tid=0x00c7db08 nid=0x3a runnable [9b27f000..9b27fc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xb75582a0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at javax.net.ssl.impl.SSLServerSocketImpHi,
I guess this issue is related to your connectivity and hyperion services. Can you take a restart and try to execute the reports. Also try to check all your Hyperion services are up and running before executing the FR Reports. You can also check the Reporting services by login to Weblogic server (if you have applied it in your environment). In weblogic it should be in Running mode and not in Admin mode. If it is not in Running mode then try to check the Data Source setting from Connection Pool in Weblogic Server.
Soumya
Maybe you are looking for
-
F110 - Automatic Payment Transactions Issue
Hi, I am working on the report program and SAPscript used in the transaction F110 - Printout/data medium. I have maintained the program and the form to be used to print the "Payment Advice" in the FBZP transaction. When i try to schedule the job, the
-
Adobe InDesign CS5.5 plugin error
Help - I have InDesign CS5.5 (version 7.5.1.304) and when I go to open a file from a client i get an error message that reads: Cannot open - Please upgrade your plug-ins to their latest versions or upgrade to the latest version of Adobe InDesign. The
-
HT4314 How do I redeem a game that as been gifted to me?
How do I redeem a game that has been gifted to me?
-
Premiere Pro CC 2014 Constantly crashing
I have been suffering constant crashes after 5 or 6 minutes of editing. I'm using blackmagic motion jpeg HD i50 (1920x1080) clips, and a timeline with the same settings. My hardware specs: Intel i7 2600k Asus P8Z68 Deluxe/Gen3 USB3 16GB RAM DDR3 1600
-
Installing Photoshop CS6 and then changing computers
Can I install CS6 on one computer and then later on (if I want to change computers) re-install it on another computer? If so, how do I port the license over to the new computer? Also, if I understand correctly, I can only have it installed on one com