Unrecoverable timeout!
Hi,
We are on SP10 and our scenario send HTTP request to marketplace.
When i send a test message to marketplace from RWB there is a timeout and the response message in sxmb_moni is :
"HTTP server code 403 reason java.io.InterruptedIOException: Unrecoverable timeout! explanation"
Is this a problem on sender side (Xi part) or on receiving side?
How can we rsolve this?
Best regards,
Laurent.
Hi francesco,
Thanks for your response but i can't create tcp gateways with the utility on my desktop. When i click 'add' button nothing happens.
i also have another error message at sending HTTP request when i execute the whole process from start:
[ISC.0049.9013] Timeout at 'Invoke for SaintGobainFunctionality.util:accessBugsE
I just don't see what this is...
Regards,
Laurent
Similar Messages
-
Wrt350n v.1.03.7 firmware dl/install
Has anyone successfully downloaded and installed the 1.03.7 firmware for the WRT350N? I keep getting time outs at around 70-80% complete when I try to get it from FTP site, which is giving me a lame 80k/sec conxn and has committed several unrecoverable timeouts over the past 3 days trying to dl this (745MB ????????????) .tgz file. I'm trying to get something (anything has to be better than the 1.03.2) to work with a linksys wusb300n card and or a dell 1490 a/b/g on a new dell latitude 830 running vista enterprise 32bit. any feedback/suggestions welcome. cheers
The file you are trying to download is the GPL source code. It contains the source code of all the software inside the router. You only want to download this file if you want to compile your own firmware.
You have to select the firmware download link and not the source code link. The latest firmware for your router in the U.S. is 1.03.2. -
Error: Halting this cluster node due to unrecoverable service failure
Our cluster has experienced some sort of fault that has only become apparent today. The origin appears to have been nearly a month ago yet the symptoms have only just manifested.
The node in question is a standalone instance running a DistributedCache service with local storage. It output the following to stdout on Jan-22:
Coherence <Error>: Halting this cluster node due to unrecoverable service failure
It finally failed today with OutOfMemoryError: Java heap space.
We're running coherence-3.5.2.jar.
Q1: It looks like this node failed on Jan-22 yet we did not notice. What is the best way to monitor node health?
Q2: What might the root cause be for such a fault?
I found the following in the logs:
2011-01-22 01:18:58,296 Coherence Logger@9216774 3.5.2/463 ERROR 2011-01-22 01:18:58.296/9910749.462 Oracle Coherence EE 3.5.2/463 <Error> (thread=Cluster, member=33): Attempting recovery (due to soft timeout) of Guard{Daemon=DistributedCache}
2011-01-22 01:18:58,296 Coherence Logger@9216774 3.5.2/463 ERROR 2011-01-22 01:18:58.296/9910749.462 Oracle Coherence EE 3.5.2/463 <Error> (thread=Cluster, member=33): Attempting recovery (due to soft timeout) of Guard{Daemon=DistributedCache}
2011-01-22 01:19:04,772 Coherence Logger@9216774 3.5.2/463 ERROR 2011-01-22 01:19:04.772/9910755.938 Oracle Coherence EE 3.5.2/463 <Error> (thread=Cluster, member=33): Terminating guarded execution (due to hard timeout) of Guard{Daemon=DistributedCache}
2011-01-22 01:19:04,772 Coherence Logger@9216774 3.5.2/463 ERROR 2011-01-22 01:19:04.772/9910755.938 Oracle Coherence EE 3.5.2/463 <Error> (thread=Cluster, member=33): Terminating guarded execution (due to hard timeout) of Guard{Daemon=DistributedCache}
2011-01-22 01:19:05,785 Coherence Logger@9216774 3.5.2/463 ERROR 2011-01-22 01:19:05.785/9910756.951 Oracle Coherence EE 3.5.2/463 <Error> (thread=Termination Thread, member=33): Full Thread Dump
Thread[Reference Handler,10,system]
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread[DistributedCache,5,Cluster]
java.nio.Bits.copyToByteArray(Native Method)
java.nio.DirectByteBuffer.get(DirectByteBuffer.java:224)
com.tangosol.io.nio.ByteBufferInputStream.read(ByteBufferInputStream.java:123)
java.io.DataInputStream.readFully(DataInputStream.java:178)
java.io.DataInputStream.readFully(DataInputStream.java:152)
com.tangosol.util.Binary.readExternal(Binary.java:1066)
com.tangosol.util.Binary.<init>(Binary.java:183)
com.tangosol.io.nio.BinaryMap$Block.readValue(BinaryMap.java:4304)
com.tangosol.io.nio.BinaryMap$Block.getValue(BinaryMap.java:4130)
com.tangosol.io.nio.BinaryMap.get(BinaryMap.java:377)
com.tangosol.io.nio.BinaryMapStore.load(BinaryMapStore.java:64)
com.tangosol.net.cache.SerializationPagedCache$WrapperBinaryStore.load(SerializationPagedCache.java:1547)
com.tangosol.net.cache.SerializationPagedCache$PagedBinaryStore.load(SerializationPagedCache.java:1097)
com.tangosol.net.cache.SerializationMap.get(SerializationMap.java:121)
com.tangosol.net.cache.SerializationPagedCache.get(SerializationPagedCache.java:247)
com.tangosol.net.cache.AbstractSerializationCache$1.getOldValue(AbstractSerializationCache.java:315)
com.tangosol.net.cache.OverflowMap$Status.registerBackEvent(OverflowMap.java:4210)
com.tangosol.net.cache.OverflowMap.onBackEvent(OverflowMap.java:2316)
com.tangosol.net.cache.OverflowMap$BackMapListener.onMapEvent(OverflowMap.java:4544)
com.tangosol.util.MultiplexingMapListener.entryDeleted(MultiplexingMapListener.java:49)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:214)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:166)
com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:556)
com.tangosol.net.cache.AbstractSerializationCache.dispatchEvent(AbstractSerializationCache.java:338)
com.tangosol.net.cache.AbstractSerializationCache.dispatchPendingEvent(AbstractSerializationCache.java:321)
com.tangosol.net.cache.AbstractSerializationCache.removeBlind(AbstractSerializationCache.java:155)
com.tangosol.net.cache.SerializationPagedCache.removeBlind(SerializationPagedCache.java:348)
com.tangosol.util.AbstractKeyBasedMap$KeySet.remove(AbstractKeyBasedMap.java:556)
com.tangosol.net.cache.OverflowMap.removeInternal(OverflowMap.java:1299)
com.tangosol.net.cache.OverflowMap.remove(OverflowMap.java:380)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$Storage.clear(DistributedCache.CDB:24)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onClearRequest(DistributedCache.CDB:32)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ClearRequest.run(DistributedCache.CDB:1)
com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheRequest.onReceived(DistributedCacheRequest.CDB:12)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:9)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:136)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onNotify(DistributedCache.CDB:3)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[Finalizer,8,system]
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread[PacketReceiver,7,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onWait(PacketReceiver.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[RMI TCP Accept-0,5,system]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
java.net.ServerSocket.implAccept(ServerSocket.java:453)
java.net.ServerSocket.accept(ServerSocket.java:421)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
java.lang.Thread.run(Thread.java:619)
Thread[PacketSpeaker,8,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketSpeaker.onNotify(PacketSpeaker.CDB:62)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[Logger@9216774 3.5.2/463,3,main]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[PacketListener1,8,Cluster]
java.net.PlainDatagramSocketImpl.receive0(Native Method)
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
java.net.DatagramSocket.receive(DatagramSocket.java:712)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[main,5,main]
java.lang.Object.wait(Native Method)
com.tangosol.net.DefaultCacheServer.main(DefaultCacheServer.java:79)
com.networkfleet.cacheserver.Launcher.main(Launcher.java:122)
Thread[Signal Dispatcher,9,system]
Thread[RMI TCP Accept-41006,5,system]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
java.net.ServerSocket.implAccept(ServerSocket.java:453)
java.net.ServerSocket.accept(ServerSocket.java:421)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
java.lang.Thread.run(Thread.java:619)
ThreadCluster
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[TcpRingListener,6,Cluster]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
java.net.ServerSocket.implAccept(ServerSocket.java:453)
java.net.ServerSocket.accept(ServerSocket.java:421)
com.tangosol.coherence.component.net.socket.TcpSocketAccepter.accept(TcpSocketAccepter.CDB:18)
com.tangosol.coherence.component.util.daemon.TcpRingListener.acceptConnection(TcpRingListener.CDB:10)
com.tangosol.coherence.component.util.daemon.TcpRingListener.onNotify(TcpRingListener.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[PacketPublisher,6,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onWait(PacketPublisher.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[RMI TCP Accept-0,5,system]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
java.net.ServerSocket.implAccept(ServerSocket.java:453)
java.net.ServerSocket.accept(ServerSocket.java:421)
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
java.lang.Thread.run(Thread.java:619)
Thread[PacketListenerN,8,Cluster]
java.net.PlainDatagramSocketImpl.receive0(Native Method)
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
java.net.DatagramSocket.receive(DatagramSocket.java:712)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[Invocation:Management,5,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[DistributedCache:PofDistributedCache,5,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[Invocation:Management:EventDispatcher,5,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[Termination Thread,5,Cluster]
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Thread.java:1487)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:791)
com.tangosol.coherence.component.net.Cluster.onServiceFailed(Cluster.CDB:5)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard.terminate(Grid.CDB:17)
com.tangosol.net.GuardSupport$2.run(GuardSupport.java:652)
java.lang.Thread.run(Thread.java:619)
2011-01-22 01:19:05,785 Coherence Logger@9216774 3.5.2/463 ERROR 2011-01-22 01:19:05.785/9910756.951 Oracle Coherence EE 3.5.2/463 <Error> (thread=Termination Thread, member=33): Full Thread Dump
Thread[Reference Handler,10,system]
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread[DistributedCache,5,Cluster]
java.nio.Bits.copyToByteArray(Native Method)
java.nio.DirectByteBuffer.get(DirectByteBuffer.java:224)
com.tangosol.io.nio.ByteBufferInputStream.read(ByteBufferInputStream.java:123)
java.io.DataInputStream.readFully(DataInputStream.java:178)
java.io.DataInputStream.readFully(DataInputStream.java:152)
com.tangosol.util.Binary.readExternal(Binary.java:1066)
com.tangosol.util.Binary.<init>(Binary.java:183)
com.tangosol.io.nio.BinaryMap$Block.readValue(BinaryMap.java:4304)
com.tangosol.io.nio.BinaryMap$Block.getValue(BinaryMap.java:4130)
com.tangosol.io.nio.BinaryMap.get(BinaryMap.java:377)
com.tangosol.io.nio.BinaryMapStore.load(BinaryMapStore.java:64)
com.tangosol.net.cache.SerializationPagedCache$WrapperBinaryStore.load(SerializationPagedCache.java:1547)
com.tangosol.net.cache.SerializationPagedCache$PagedBinaryStore.load(SerializationPagedCache.java:1097)
com.tangosol.net.cache.SerializationMap.get(SerializationMap.java:121)
com.tangosol.net.cache.SerializationPagedCache.get(SerializationPagedCache.java:247)
com.tangosol.net.cache.AbstractSerializationCache$1.getOldValue(AbstractSerializationCache.java:315)
com.tangosol.net.cache.OverflowMap$Status.registerBackEvent(OverflowMap.java:4210)
com.tangosol.net.cache.OverflowMap.onBackEvent(OverflowMap.java:2316)
com.tangosol.net.cache.OverflowMap$BackMapListener.onMapEvent(OverflowMap.java:4544)
com.tangosol.util.MultiplexingMapListener.entryDeleted(MultiplexingMapListener.java:49)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:214)
com.tangosol.util.MapEvent.dispatch(MapEvent.java:166)
com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:556)
com.tangosol.net.cache.AbstractSerializationCache.dispatchEvent(AbstractSerializationCache.java:338)
com.tangosol.net.cache.AbstractSerializationCache.dispatchPendingEvent(AbstractSerializationCache.java:321)
com.tangosol.net.cache.AbstractSerializationCache.removeBlind(AbstractSerializationCache.java:155)
com.tangosol.net.cache.SerializationPagedCache.removeBlind(SerializationPagedCache.java:348)
com.tangosol.util.AbstractKeyBasedMap$KeySet.remove(AbstractKeyBasedMap.java:556)
com.tangosol.net.cache.OverflowMap.removeInternal(OverflowMap.java:1299)
com.tangosol.net.cache.OverflowMap.remove(OverflowMap.java:380)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$Storage.clear(DistributedCache.CDB:24)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onClearRequest(DistributedCache.CDB:32)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ClearRequest.run(DistributedCache.CDB:1)
com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheRequest.onReceived(DistributedCacheRequest.CDB:12)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:9)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:136)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onNotify(DistributedCache.CDB:3)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[Finalizer,8,system]
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread[PacketReceiver,7,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onWait(PacketReceiver.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[RMI TCP Accept-0,5,system]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
java.net.ServerSocket.implAccept(ServerSocket.java:453)
java.net.ServerSocket.accept(ServerSocket.java:421)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
java.lang.Thread.run(Thread.java:619)
Thread[PacketSpeaker,8,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketSpeaker.onNotify(PacketSpeaker.CDB:62)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[Logger@9216774 3.5.2/463,3,main]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[PacketListener1,8,Cluster]
java.net.PlainDatagramSocketImpl.receive0(Native Method)
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
java.net.DatagramSocket.receive(DatagramSocket.java:712)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[main,5,main]
java.lang.Object.wait(Native Method)
com.tangosol.net.DefaultCacheServer.main(DefaultCacheServer.java:79)
com.networkfleet.cacheserver.Launcher.main(Launcher.java:122)
Thread[Signal Dispatcher,9,system]
Thread[RMI TCP Accept-41006,5,system]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
java.net.ServerSocket.implAccept(ServerSocket.java:453)
java.net.ServerSocket.accept(ServerSocket.java:421)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
java.lang.Thread.run(Thread.java:619)
ThreadCluster
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[TcpRingListener,6,Cluster]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
java.net.ServerSocket.implAccept(ServerSocket.java:453)
java.net.ServerSocket.accept(ServerSocket.java:421)
com.tangosol.coherence.component.net.socket.TcpSocketAccepter.accept(TcpSocketAccepter.CDB:18)
com.tangosol.coherence.component.util.daemon.TcpRingListener.acceptConnection(TcpRingListener.CDB:10)
com.tangosol.coherence.component.util.daemon.TcpRingListener.onNotify(TcpRingListener.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[PacketPublisher,6,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onWait(PacketPublisher.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[RMI TCP Accept-0,5,system]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
java.net.ServerSocket.implAccept(ServerSocket.java:453)
java.net.ServerSocket.accept(ServerSocket.java:421)
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
java.lang.Thread.run(Thread.java:619)
Thread[PacketListenerN,8,Cluster]
java.net.PlainDatagramSocketImpl.receive0(Native Method)
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
java.net.DatagramSocket.receive(DatagramSocket.java:712)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:619)
Thread[Invocation:Management,5,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[DistributedCache:PofDistributedCache,5,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[Invocation:Management:EventDispatcher,5,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:619)
Thread[Termination Thread,5,Cluster]
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Thread.java:1487)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:791)
com.tangosol.coherence.component.net.Cluster.onServiceFailed(Cluster.CDB:5)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard.terminate(Grid.CDB:17)
com.tangosol.net.GuardSupport$2.run(GuardSupport.java:652)
java.lang.Thread.run(Thread.java:619)
2011-01-22 01:19:06,738 Coherence Logger@9216774 3.5.2/463 INFO 2011-01-22 01:19:06.738/9910757.904 Oracle Coherence EE 3.5.2/463 <Info> (thread=main, member=33): Restarting Service: DistributedCache
2011-01-22 01:19:06,738 Coherence Logger@9216774 3.5.2/463 INFO 2011-01-22 01:19:06.738/9910757.904 Oracle Coherence EE 3.5.2/463 <Info> (thread=main, member=33): Restarting Service: DistributedCache
2011-01-22 01:19:06,738 Coherence Logger@9216774 3.5.2/463 ERROR 2011-01-22 01:19:06.738/9910757.904 Oracle Coherence EE 3.5.2/463 <Error> (thread=main, member=33): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: Distr
butedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=16, BackupPartitions=16}
2011-01-22 01:19:06,738 Coherence Logger@9216774 3.5.2/463 ERROR 2011-01-22 01:19:06.738/9910757.904 Oracle Coherence EE 3.5.2/463 <Error> (thread=main, member=33): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: Distr
butedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=16, BackupPartitions=16}Hi
It seems like the problem in this case is the call to clear() which will try to load all entries stored in the overflow scheme to emit potential cache events to listeners. This probably requires much more memory than there is Java heap available, hence the OOM.
Our recommendation in this case is to call destroy() since this will bypass the even firing.
/Charlie -
Resource start timeout always at 60 seconds
Hello all,
I'm using clusterware for a standard database cluster. As my storage is NetApp I want to use snapdrive software to load dynamically LUN (3 per database). I've a script correctly working by hand. Time needed for LUNs to be attached is variable between 80 and 120 seconds. Resource is called database.ofa.
crs_profile -print database.ofa
NAME=database.ofa
TYPE=application
ACTION_SCRIPT=/app/oracle/product/crs/crs/public/act_snapdrive.pl
ACTIVE_PLACEMENT=0
AUTO_START=restore
CHECK_INTERVAL=450
DESCRIPTION=mantprd.ofa
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=
OPTIONAL_RESOURCES=
PLACEMENT=balanced
REQUIRED_RESOURCES=database
RESTART_ATTEMPTS=1
SCRIPT_TIMEOUT=450
START_TIMEOUT=0
STOP_TIMEOUT=0
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=mantprd
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=
Starting manually resource will fail. crsd.log reports:
2009-03-24 16:01:23.538: [ CRSRES][1510144352]0Attempting to start `database.ofa` on member `server01`
2009-03-24 16:02:26.537: [ CRSEVT][1510144352]0CAAMonitorHandler :: 0:Could not join /app/oracle/product/crs/crs/public/act_snapdrive.pl(start)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
2009-03-24 16:02:26.537: [ CRSEVT][1510144352]0CAAMonitorHandler :: 0:Action Script /app/oracle/product/crs/crs/public/act_snapdrive.pl(start) timed out for database.ofa! (timeout=60)
2009-03-24 16:02:26.537: [ CRSAPP][1510144352]0StartResource error for database.ofa error code = -2
2009-03-24 16:03:29.559: [ CRSEVT][1510144352]0CAAMonitorHandler :: 0:Could not join /app/oracle/product/crs/crs/public/act_snapdrive.pl(stop)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
2009-03-24 16:03:29.559: [ CRSEVT][1510144352]0CAAMonitorHandler :: 0:Action Script /app/oracle/product/crs/crs/public/act_snapdrive.pl(stop) timed out for database.ofa! (timeout=60)
2009-03-24 16:03:29.559: [ CRSAPP][1510144352]0StopResource error for database.ofa error code = -2
2009-03-24 16:03:29.572: [ CRSRES][1510144352]0X_OP_StopResourceFailed : Stop Resource failed
(File: rti.cpp, line: 1803
2009-03-24 16:03:29.573: [ CRSRES][1510144352][ALERT]0`database.ofa` on member `server01` has experienced an unrecoverable failure.
2009-03-24 16:03:29.573: [ CRSRES][1510144352]0Human intervention required to resume its availability.
Seems that timeout reported is always 60seconds I've manually changed for all resources options st=450 as shown above but always crs fails at 60 seconds.
Any hints to debug?
Thanks
StefanoHi,
for: "+I didn't run crs_register to update resource. Using crs_profile is not enough?+": YES, crs_profile creates a profile (only). You need to register the resource using crs_register to make it manageable by Oracle Clusterware:
[CLUSTERWARE]> crs_profile -create MyProfTest -t application
[CLUSTERWARE]> crs_stat -t |grep MyProfTest
[CLUSTERWARE]> touch /ora/clusterware/crs/public/MyProfTest.scr
[CLUSTERWARE]> crs_register MyProfTest
[CLUSTERWARE]> crs_stat -t |grep MyProfTest
MyProfTest application OFFLINE OFFLINE
Concluding, if crs_profile is used on a resource that has been registered already, the changes would only change the profile. To make the changes take effect, you would need to re-register the resource with crs_register -update. Now, if it is only a minor update, you can actually use crs_register directly and change the attribute that needs to be changed and re-register the resource in one go:
Usage: crs_register resource_name [-dir directory_path] [...] [-u] [-f] [-q]
crs_register resource_name -update [-a action_script]
[-d description] [-h hosting_members] [-r required_resources]
[-l optional_resources] [-p placement_policy]
[-o as=auto_start,ci=check_interval,ft=failure_threshold,
fi=failure_interval,ra=restart_attempts,fd=failover_delay,
st=script_timeout,ap=active_placement,
ut=uptime_threshold,*rt=start_timeout*,pt=stop_timeout] [-q]
Hope that helps. Thanks,
Markus -
Dealing with hard timeout of guarded service
Hi, I'm investigating the behavior the cause of and subsequent behavior after a hard timeout of a guarded service. In my experience, the grid members are unable to recover properly. I am trying to figure out whether there is something in our configuration that may be aggravating the situation, and also see whether I might be able to improve the behavior of our client code.
What I have done is purposely lower the service guardian's timeout, to about 25 seconds, so that a certain EntryProcessor will always time-out. The behavior I see after the timeout is very similar to the behavior we see when the problem crops up in the real world. I know that I can raise the timeout or use the PriorityTask API, but even if I do that, we may run into timeouts due to deadlock. If that happens, I want to know what I can expect the grid to do.
Here's what I'm seeing:
I am running 2 storage-enabled members with backup-count=0. I am running my "client" as a non-storage-enabled member.
First, the client kicks off the EntryProcessor:
Map<Object, Integer> readProcessorResult = requestCache.invokeAll(AlwaysFilter.INSTANCE, processor);
The two members begin logging successful executions of the EntryProcessor. Then, one of the members happens to run it against an entry that takes longer than the service guardian's timeout. We get a soft and hard timeout:
[ERROR] Coherence: 2012-01-26 10:56:20.103/333.297Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=1): Detected soft timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:jdw-read-request-service} Service=PartitionedCache{Name=jdw-read-request-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=0, AssignedPartitions=128}}
[WARN ] Coherence: 2012-01-26 10:56:20.121/333.315Oracle Coherence GE 3.7.1.2 <Warning> (thread=Recovery Thread, member=1): Attempting recovery of Guard{Daemon=DistributedCache:jdw-read-request-service}
[ERROR] Coherence: 2012-01-26 10:56:22.604/335.798Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=1): Detected hard timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:jdw-read-request-service} Service=PartitionedCache{Name=jdw-read-request-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=0, AssignedPartitions=128}}Now, for some reason (why?), Coherence determines that the service is unrecoverable and it must stop the cluster:
[WARN ] Coherence: 2012-01-26 10:56:22.605/335.799Oracle Coherence GE 3.7.1.2 <Warning> (thread=Termination Thread, member=1): Terminating Guard{Daemon=DistributedCache:jdw-read-request-service}
Coherence <Error>: Halting this cluster node due to unrecoverable service failure
[ERROR] Coherence: 2012-01-26 10:56:23.613/336.807Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=1): StopRunning ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1} due to unhandled exception:
[ERROR] Coherence: 2012-01-26 10:56:23.613/336.807Oracle Coherence GE 3.7.1.2 <Error> (thread=PacketListener1P, member=1): Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: UdpSocket.receive: unable to reopen socket; State=STATE_CLOSED
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:58)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Unknown Source)
[ERROR] Coherence: 2012-01-26 10:56:23.641/336.835Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=n/a):
java.lang.NullPointerException: null
at com.tangosol.coherence.component.net.Cluster$ClusterService$TcpRing.onAcceptException(Cluster.CDB:13) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.net.TcpRing.onAccept(TcpRing.CDB:25) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.net.TcpRing.onSelect(TcpRing.CDB:27) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.net.TcpRing.select(TcpRing.CDB:14) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onWait(ClusterService.CDB:6) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39) ~[coherence-3.7.1.2.jar:3.7.1.2]
at java.lang.Thread.run(Unknown Source) [na:1.6.0_30]
[WARN ] Coherence: 2012-01-26 10:56:24.976/338.170Oracle Coherence GE 3.7.1.2 <Warning> (thread=Invocation:jdw-invocation-service, member=n/a): failed to stop 95 worker threads; abandoning
Coherence <Error>: Halted the cluster:
Cluster is not running: State=5
Exception in thread "Cluster|SERVICE_STOPPED|Member(Id=1, Timestamp=2012-01-26 10:50:58.044, Address=192.168.1.67:9001, MachineId=52295, Location=site:,machine:DEN12956L,process:10012, Role=CoherenceServer)" java.nio.channels.ClosedSelector
Exception
at sun.nio.ch.SelectorImpl.keys(Unknown Source)
at com.tangosol.coherence.component.net.TcpRing.disconnectAll(TcpRing.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService$TcpRing.onLeft(ClusterService.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onStopRunning(ClusterService.CDB:7)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onException(ClusterService.CDB:28)
at com.tangosol.coherence.component.net.Cluster$ClusterService.onException(Cluster.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:85)
[ERROR] Coherence: 2012-01-26 10:56:25.616/338.810 Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=n/a): StopRunning ClusterService{Name=Cluster, State=(SERVICE_STOPPED, STATE_JOINED), Id=0, Version=3.7.1} due to unhandled exception:
at java.lang.Thread.run(Unknown Source)
[ERROR] Coherence: 2012-01-26 10:56:25.616/338.810Oracle Coherence GE 3.7.1.2 <Error> (thread=Cluster, member=n/a):
java.nio.channels.ClosedSelectorException: null
at sun.nio.ch.SelectorImpl.keys(Unknown Source) ~[na:1.6.0_30]
at com.tangosol.coherence.component.net.TcpRing.close(TcpRing.CDB:11) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onExit(ClusterService.CDB:1) ~[coherence-3.7.1.2.jar:3.7.1.2]
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:68) ~[coherence-3.7.1.2.jar:3.7.1.2]
at java.lang.Thread.run(Unknown Source) [na:1.6.0_30]Around the same time that member 1 decided to stop its cluster, member 2, which had been happily executing entry processors, begins taking responsibility for member 1's partitions. That's what I'd expect.
[WARN ] Coherence: 2012-01-26 10:56:23.643/336.784Oracle Coherence GE 3.7.1.2 <Warning> (thread=DistributedCache:sys-id-dist-service, member=2): Assigned 128 orphaned primary partitions
[WARN ] Coherence: 2012-01-26 10:56:23.646/336.787Oracle Coherence GE 3.7.1.2 <Warning> thread=DistributedCache:sourceMetadataReviewCache-service, member=2): Assigned 128 orphaned primary partitions
......Member 1 now restarts its cluster and re-joins as member 4, and starts re-starting its services:
[INFO ] Coherence: 2012-01-26 10:56:26.008/339.202Oracle Coherence GE 3.7.1.2 <Info> (thread=main, member=n/a): Restarting cluster
[INFO ] Coherence: 2012-01-26 10:56:26.327/339.521Oracle Coherence GE 3.7.1.2 <Info> (thread=Cluster, member=n/a): This Member(Id=4, Timestamp=2012-01-26 10:56:26.126, Address=192.168.1.67:9001, MachineId=52295, Location=site:,machine:DEN12956L,process:10012, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) joined cluster "NIR_GRID_DEV" with senior Member(Id=2, Timestamp=2012-01-26 10:51:03.593, Address=192.168.1.67:9003, MachineId=52295, Location=site:,machine:DEN12956L,process:10024, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
[INFO ] Coherence: 2012-01-26 10:56:26.364/339.558Oracle Coherence GE 3.7.1.2 <Info> (thread=main, member=n/a): Started cluster Name=NIR_GRID_DEV
Group{Address=224.3.7.0, Port=60000, TTL=0}
MasterMemberSet(
ThisMember=Member(Id=4, Timestamp=2012-01-26 10:56:26.126, Address=192.168.1.67:9001, MachineId=52295, Location=site:,machine:DEN12956L,process:10012, Role=CoherenceServer)
OldestMember=Member(Id=2, Timestamp=2012-01-26 10:51:03.593, Address=192.168.1.67:9003, MachineId=52295, Location=site:,machine:DEN12956L,process:10024, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=3
Member(Id=2, Timestamp=2012-01-26 10:51:03.593, Address=192.168.1.67:9003, MachineId=52295, Location=site:,machine:DEN12956L,process:10024, Role=CoherenceServer)
Member(Id=3, Timestamp=2012-01-26 10:55:05.522, Address=192.168.1.67:9005, MachineId=52295, Location=site:,machine:DEN12956L,process:13268, Role=IntellijRtExecutionAppMain)
Member(Id=4, Timestamp=2012-01-26 10:56:26.126, Address=192.168.1.67:9001, MachineId=52295, Location=site:,machine:DEN12956L,process:10012, Role=CoherenceServer)
MemberId|ServiceVersion|ServiceJoined|MemberState
2|3.7.1|2012-01-26 10:51:03.593|JOINED,
3|3.7.1|2012-01-26 10:55:05.522|JOINED,
4|3.7.1|2012-01-26 10:56:26.337|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
TcpRing{Connections=[2, 3]}
IpMonitor{AddressListSize=0}
[INFO ] Coherence: 2012-01-26 10:56:26.365/339.559Oracle Coherence GE 3.7.1.2 <Info> (thread=main, member=4): Restarting Service:Management
[INFO ] Coherence: 2012-01-26 10:56:26.417/339.611Oracle Coherence GE 3.7.1.2 <Info> (thread=main, member=4): Restarting Service:jdwSourceManagerCache-service
......While that is happening, member 2 now also encounters a hard timeout on the same service and begins going through the same process that member 1 just went through. I am not sure why member 2 encounters this timeout. Perhaps it is because it ran another entry processor that took too long? It is difficult to tell. Except this time we start having problems when member 2 tries to stop the cluster, and I'm not sure exactly why. Perhaps because member 4 is not yet in a good state?
[ERROR] Coherence: 2012-01-26 10:56:35.282/348.423Oracle Coherence GE 3.7.1.2 <Error> (thread=PacketListener1, member=2): Stopping cluster due to unhandled exception:
com.tangosol.net.messaging.ConnectionException: UdpSocket.receive: unable to reopen socket; State=STATE_CLOSED
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:58)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Unknown Source)
Coherence <Error>: Halted the cluster:
Cluster is not running: State=5
[ERROR] Coherence: 2012-01-26 10:56:42.147/355.288Oracle Coherence GE 3.7.1.2 <Error> (thread=main, member=n/a): Failed to restart services: com.tangosol.net.RequestTimeoutException: Timeout while waiting for cluster to stop.
[ERROR] Coherence: 2012-01-26 10:56:51.324/364.465Oracle Coherence GE 3.7.1.2 <Error> (thread=main, member=n/a): Failed to restart services: com.tangosol.net.RequestTimeoutException: Timeout while waiting for cluster to stop.
...... (error repeats)And the newly re-joined member 4, which had been restarting services and claiming partitions from the newly-dead member 2, appears to restart execution of the entry processor but then complains that its thread was interrupted: I am not sure why this happens or what it means.
[INFO ] Coherence: 2012-01-26 10:56:52.685/365.879Oracle Coherence GE 3.7.1.2 <Info> (thread=DistributedCache:jdw-read-request-service, member=4): Restarting NamedCache: coherence.common.sequencegenerators
[ERROR] Coherence: 2012-01-26 10:56:52.686/365.880Oracle Coherence GE 3.7.1.2 <Error> (thread=DistributedCache:jdw-read-request-service, member=4): This thread was interrupted while waiting for the results of a request:
Poll
PollId=2, active
InitTimeMillis=1327597012685
Service=DistributedCacheForSequenceGenerators (60)
RespondedMemberSet=[]
LeftMemberSet=[]
RemainingMemberSet=[4]
},Meanwhile, my client, who ran the invoke() with the EntryProcessor, has received a message stating "no storage enabled members exist," which although it may be true doesn't seem like it's really the cause of the problem.
Could I be having problems because all my cluster members are executing the same kind of long-running EntryProcessor and all of them are getting service guardian timeouts at around the same time? How do I deal with that? For example, what if a database is running very slowly one day and my entry processors all start to execute slowly? I don't want my whole grid having problems. Should I, for example, limit the number of concurrent EntryProcessors to < count of grid members by using a LimitFilter?
What does it mean that there was a timeout waiting for the cluster to stop?
And what about the client? If I get an exception after running a grid command, is it valid to wait a few seconds for things to stabilize and then try again? What's my best bet as far as that goes?
Thanks! And let me know if I should file a support request instead.rehevkor5 wrote:
I appreciate the responses so far, but unfortunately they give me no new information.
I am aware that I can try to avoid timeouts by using the PriorityTask API and heartbeats, but that still gives me no guarantee that I will not run into a deadlock situation and encounter a timeout. That is fine as long as my grid can deal with it. Currently, it does not behave well when that happens (all members eventually die). I challenge anyone reading this post to try running an entry processor on their grid that is designed to time out (not by using sleep() as that is interruptable via a soft timeout). All your members will probably encounter the timeout at the same time, and it probably will not end well. If, however, you can handle this situation or have some approach that mitigates it, I would love to hear from you!
I am also aware that I can disable the service guardians, or configure them to only log and take no action. However, this leaves me vulnerable to deadlocks.
Therefore, I would still appreciate any responses that address either of these two issues:
1. Configuration changes or other fixes that allow my members to recover successfully from a service timeout
2. Operational best practices that allow my grid to continue running even if an entry processor that is running on every member of the grid times out at the same time. For example, limiting the number of concurrent entry processors.One operational best practice is design your system so you can't get into a deadlock. If it can get into a deadlock once, it can get into it many times. If a system does that, it is not a well-behaving or well-designed system.
If your code can get into a deadlock across the cluster, that usually means that you are trying to do something which you should not have, like trying to operate on multiple resources in an arbitrary order while holding locks on them (e.g. violating the threading guidelines), which is the typical case of a distributed deadlock, and against which you can guard with ordering your multiple locking operations in a consistent order, but you should not expect Coherence to retry and possibly get into the same problem, as all you would achieve with it is that you converted the deadlock into a livelock.
Coherence is not supposed to guard against your errors. With the guardian it gives you a way to find out what your error is, but it should not attempt the impossible by trying to correct it. You should design your system so that it does not have this kinds of errors.
If you think there is a specific reproducible problem in how Coherence reacts to your code which can bring down the system, then create a test case and submit it in a SR. The kinds of exceptions you get make it possible that it may be necessary. If it is indeed Coherence's fault that it dies and not something you did, then I am fairly certain that it will be given quite high priority considering it is an error which can bring a cluster down.
Coherence server-side constructs have an API contract. Single foremost aspect of that is that the operations complete without an error (unless otherwise documented) within a small period of time. If it goes against this then it has to provide feedback or customization with PriorityTask/GuardSupport. A code which can deadlock goes against this contract. After all we are speaking about a grid, not a distributed process manager.
I agree that the grid should not die due to a deadlock, but you should not expect Coherence to fix a deadlock either. But dying due to an exception and having a distributed deadlock is two different and independent issues.
Best regards,
Robert
Edited by: robvarga on Feb 3, 2012 10:12 AM -
Sso session timeout per partner application
Hello,
I was just wondering if it is possible to configure SSO session timeouts per partner application? I'm looking to log out users of a particular application after 15 minutes, but don't want this change to affect any of my other SSO enabled applications. Is this possible?
Thanks,Hi,
I do not think so, you can not specify specail parameter for one application in SSO.
Why because SSO is one component (within your Infra) through which you logon different apps.
Another solution may be it will expensive is that you 'll need to use different infra for this specific application.
Regards,
Hamdy -
I am using the mac QQ and when I login it said login timeout.
I am using the mac QQ and when I login it said login timeout.
If you are missing using google maps - try the Nokia map app called "here"
-
Dear all,
I inherit this configuration from my colleague,
The PC / host inside the network internet connection will timeout / disconnected after several minutes when not using.
How do i disable the config and I want the host to continously connect to internet.
=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2011.08.11 11:01:19 =~=~=~=~=~=~=~=~=~=~=~=
kewpie-MLK-ASA# sh run
: Saved
ASA Version 8.0(3)
hostname kewpie-MLK-ASA
domain-name default.domain.invalid
enable password ym1CwmrLnc/fndsu encrypted
names
interface Ethernet0/0
nameif outside
security-level 0
ip address 60.a.a.54 255.255.255.252
interface Ethernet0/1
no nameif
no security-level
no ip address
interface Ethernet0/1.1
vlan 10
nameif Inside
security-level 80
ip address 192.168.0.1 255.255.255.0
interface Ethernet0/1.2
vlan 20
nameif visitor
security-level 100
ip address 192.168.1.1 255.255.255.0
interface Ethernet0/2
shutdown
no nameif
no security-level
no ip address
interface Ethernet0/3
shutdown
no nameif
no security-level
no ip address
interface Management0/0
shutdown
no nameif
no security-level
no ip address
passwd 2KFQnbNIdI.2KYOU encrypted
ftp mode passive
dns server-group DefaultDNS
domain-name default.domain.invalid
access-list 100 extended permit icmp any any
access-list 100 extended permit tcp any any
access-list 100 extended permit ip any any
access-list 101 extended permit icmp any any
access-list 101 extended permit tcp any any eq 2828
access-list 101 extended permit tcp any host 192.168.0.254 eq 2255
pager lines 24
mtu outside 1500
mtu Inside 1500
mtu visitor 1500
icmp unreachable rate-limit 1 burst-size 1
icmp permit any outside
icmp permit any Inside
icmp permit any visitor
asdm image disk0:/asdm-507.bin
no asdm history enable
arp timeout 14400
global (outside) 1 interface
nat (Inside) 1 192.168.0.0 255.255.255.0
nat (visitor) 1 192.168.1.0 255.255.255.0
static (Inside,outside) tcp interface 2828 192.168.0.254 telnet netmask 255.255.255.255
access-group 101 in interface outside
access-group 100 in interface Inside
access-group 100 in interface visitor
route outside 0.0.0.0 0.0.0.0 60.a.a.53 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout uauth 0:30:00 absolute uauth 0:30:00 inactivity
dynamic-access-policy-record DfltAccessPolicy
aaa authentication include tcp/0 Inside 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 LOCAL
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
telnet 0.0.0.0 0.0.0.0 Inside
telnet 192.168.4.0 255.255.255.0 Inside
telnet timeout 5
ssh 0.0.0.0 0.0.0.0 outside
ssh 0.0.0.0 0.0.0.0 Inside
ssh timeout 5
console timeout 0
dhcpd dns 202.188.0.133 202.188.5.1
dhcpd address 192.168.0.2-192.168.0.253 Inside
dhcpd enable Inside
dhcpd address 192.168.1.2-192.168.1.253 visitor
dhcpd enable visitor
threat-detection basic-threat
threat-detection statistics access-list
username admin password bOnxO8/ZA7i5hOxq encrypted
username kpmsb password /LTd0pEXjM6Ht1Sp encrypted
class-map inspection_default
match default-inspection-traffic
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect esmtp
inspect sqlnet
inspect sunrpc
inspect tftp
inspect sip
inspect xdmcp
service-policy global_policy global
prompt hostname context
Cryptochecksum:809895a4506cb7e47a57552c4a0e0a0f
: endHi Mohammad,
You have the following timeoutr values set:
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout uauth 0:30:00 absolute uauth 0:30:00 inactivity
If you do not want the connection to timeout, use the following:
timeout conn 0:00:00
This would never timeout the connection.
Thanks,
Varun -
Jackd + guitar: "timeouts and broken pipes"
Hi friends! I'm trying to pass my electric guitar via any rack/effects (like Guitarix or Creox) with no luck. I've got this sound card:
01:06.0 Multimedia audio controller: Creative Labs [SB Live! Value] EMU10k1X
01:06.1 Input device controller: Creative Labs [SB Live! Value] Input device controller
I try with QJackCtl and invoking jackd from the terminal with any luck.
jackd -d alsa -C -P
jackd 0.121.3
Copyright 2001-2009 Paul Davis, Stephane Letz, Jack O'Quinn, Torben Hohn and others.
jackd comes with ABSOLUTELY NO WARRANTY
This is free software, and you are welcome to redistribute it
under certain conditions; see the file COPYING for details
could not open driver .so '/usr/lib/jack/jack_net.so': libcelt0.so.2: cannot open shared object file: No such file or directory
could not open driver .so '/usr/lib/jack/jack_firewire.so': libffado.so.2: cannot open shared object file: No such file or directory
JACK compiled with System V SHM support.
loading driver ..
creating alsa driver ... hw:0|hw:0|1024|2|48000|0|0|nomon|swmeter|-|32bit
control device hw:0
configuring for 48000Hz, period = 1024 frames (21.3 ms), buffer = 2 periods
ALSA: final selected sample format for capture: 16bit little-endian
ALSA: use 2 periods for capture
ALSA: final selected sample format for playback: 16bit little-endian
ALSA: use 2 periods for playback
jackd watchdog: timeout - killing jackd
[gabo@machina ~]$
This is the output from QJackCtl:
00:12:07.126 Client deactivated.
00:12:07.130 JACK is being forced...
cannot read server event (Success)
cannot continue execution of the processing graph (Bad file descriptor)
zombified - calling shutdown handler
cannot send request type 7 to server
cannot read result for request type 7 from server (Broken pipe)
cannot send request type 7 to server
cannot read result for request type 7 from server (Broken pipe)
00:12:07.339 JACK was stopped with exit status=1.
I can hear my guitar and record with Audacity, but when jackd enter into the escenario everything blows up. I read that nowadays almost any sound card will work with QJackCtl with the default options. I play with the parameters and sometimes jack refuse to start. With the default options on i can make it run, but i get no sound of the racks or guitar effects processors neither the guitar tuners that use jack takes the sound from the guitar. My line input is in capture via alsamixer, but still no luck. Any clue on this? I'm skipping steps?
Thanks in advance.
iamgaboHi!
groups && cat /proc/asound/cards && cat ~/.asoundrc && cat '/etc/security/limits.d/audio.conf' && jackd -v
adm disk lp wheel http network video audio optical storage power users polkitd vboxusers wireshark kismet
0 [Live ]: EMU10K1X - Dell Sound Blaster Live!
Dell Sound Blaster Live! at 0xcc00 irq 17
#pcm.upmix71 {
#type upmix
#slave.pcm "surround71"
#delay 15
#channels 8
pcm.!default {
type hw
card 0
ctl.!default {
type hw
card 0
# convert alsa API over jack API
# use it with
# % aplay foo.wav
# use this as default
pcm.!default {
type plug
slave { pcm "jack" }
ctl.mixer0 {
type hw
card 1
# pcm type jack
pcm.jack {
type jack
playback_ports {
0 system:playback_1
1 system:playback_2
capture_ports {
0 system:capture_1
1 system:capture_2
cat: /etc/security/limits.d/audio.conf: No such file or directory
I have a file called 99-audio.conf
cat /etc/security/limits.d/99-audio.conf
@audio - rtprio 99
@audio - memlock unlimited
Also i've seen some guys changing this file too:
cat /etc/security/limits.conf
# /etc/security/limits.conf
#Each line describes a limit for a user in the form:
#<domain> <type> <item> <value>
#Where:
#<domain> can be:
# - an user name
# - a group name, with @group syntax
# - the wildcard *, for default entry
# - the wildcard %, can be also used with %group syntax,
# for maxlogin limit
#<type> can have the two values:
# - "soft" for enforcing the soft limits
# - "hard" for enforcing hard limits
#<item> can be one of the following:
# - core - limits the core file size (KB)
# - data - max data size (KB)
# - fsize - maximum filesize (KB)
# - memlock - max locked-in-memory address space (KB)
# - nofile - max number of open files
# - rss - max resident set size (KB)
# - stack - max stack size (KB)
# - cpu - max CPU time (MIN)
# - nproc - max number of processes
# - as - address space limit (KB)
# - maxlogins - max number of logins for this user
# - maxsyslogins - max number of logins on the system
# - priority - the priority to run user process with
# - locks - max number of file locks the user can hold
# - sigpending - max number of pending signals
# - msgqueue - max memory used by POSIX message queues (bytes)
# - nice - max nice priority allowed to raise to values: [-20, 19]
# - rtprio - max realtime priority
#<domain> <type> <item> <value>
#* soft core 0
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#@student - maxlogins 4
* - rtprio 0
* - nice 0
@audio - rtprio 65
@audio - nice -10
@audio - memlock unlimited
jackd 0.121.3
There are the snaps for QJackCtl
Also, checkout this stuff that i've recorded with audacity, only from the line and nothing else
http://ompldr.org/vZ3A2eg
Thanks!
Last edited by iamgabo (2012-12-15 02:21:08) -
Database connection timeouts and datasource errors
Connections in the pool randomly die overnight. Stack traces show that for some reason, the evermind driver is being used even though the MySql connection pool is specified.
Also, the evermind connection pool is saying connections aren't being closed, and the stack trace shows they're being allocated by entity beans that are definitely not left hanging around.
Sometimes we get non-serializable errors when trying to retrieve the datasource (this is only after the other errors start). Some connections returned from the pool are still good, so the application limps along.
EJBs and DAOs both use jdbc/SQLServerDSCore.
Has anyone seen this problem?
<data-sources>
<data-source
class="com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource"
name="SQLServerDSCore"
location="jdbc/SQLServerDSCore"
xa-location="jdbc/xa/SQLServerXACore"
ejb-location="jdbc/SQLServerDSCore"
connection-driver="com.mysql.jdbc.Driver"
min-connections="5"
username="xxx"
password="xxx"
staleness-timeout="3600"
alive-poll-query="SELECT 1 FROM medispan"
url="jdbc:mysql://1.2.3.4:3306/dbo?autoReconnect=true&autoReconnectForPools=true&cachePrepStmts=true&is-connection-validation-required=true"
inactivity-timeout="30"
>
<property name="autoReconnect" value="true"/>
<property name="autoReconnectForPools" value="true"/>
<property name="is-connection-validation-required" value="true"/>
<property name="cachePrepStmts" value="true"/>
</data-source>
</data-sources>Rick,
OC4J 9.0.4.0.0 - BTW, do you know of any patches?As far as I know, there are no patches for the 9.0.4
production version of OC4J stand-alone.
I'm using container managed persistence,It was not clear to me, from your previous post, that you
are using CMP entity beans.
I found staleness-timeout and alive-poll-query
somewhere on a website when trying to track this
down. Here's four sources:Those sources refer to OrionServer -- and an older version, too, it seems.
Like all other Oracle products that start out as somebody
else's -- including, for example, JBuilder (that became "JDeveloper"), Apache Web Server (that became "Oracle HTTP Server") and TopLink -- their development paths diverge, until, eventually, there is absolutely no similarity between them at all. Hence, the latest versions of OC4J and "OrionServer" are so different, that you cannot be sure that something that works for "OrionServer" will work for OC4J.
I recall reading something, somewhere, sometime about configuring OC4J to use different databases (other than Oracle), but I really don't remember any details (since it was not relevant to me, because we only use Oracle database). In any case, it is possible to use a non-Oracle database with OC4J.
Good Luck,
Avi. -
MDB Timeouts and transaction behavior
Hi, thanks in advance for any help with this.
We have a MDB where we have set the timeout to five minutes. In a particular case this timeout is being reached but even though the MDB times out the transaction and sends the message for redelivery the original transaction that was processing the message continues until something happens from an application standpoint to cause that original transaction to complete. This means that we have the same message processed twice under these circumstances.
I would have expected that if a transaction timed out that the transaction would have been terminated and therefore our processing would have stopped. Is there a way to force this behavior or do we have to put an alarm in our code and kill ourselves such that we never encounter the transaction time out?
Thanks,
Sue ShanabrookI should have mentioned that we are using container managed transactions.
-
Lock Timeouts and Heap Space Exhaustion
I'm having some trouble figuring out the best way to handle resource constraints in my application. Generally speaking, the application works well after starting for about a day or so, but inevitably starts generating "Lock timeout" messages and eventually runs out of heap space.
Here is the main entity class:
http://github.com/justindthomas/flower_as/blob/master/src/java/name/justinthomas/flower/analysis/statistics/StatisticalInterval.java
Here is a supporting, persistent class:
http://github.com/justindthomas/flower_as/blob/master/src/java/name/justinthomas/flower/analysis/statistics/StatisticalFlow.java
This is the data accessor:
http://github.com/justindthomas/flower_as/blob/master/src/java/name/justinthomas/flower/analysis/statistics/StatisticsAccessor.java
And here is the class that controls the insertion and retrieval of data:
http://github.com/justindthomas/flower_as/blob/master/src/java/name/justinthomas/flower/analysis/statistics/StatisticsManager.java
The general flow is that a netflow packet is received from a sensor and the StatisticsManager normalizes the flow as it is inserted into the database. The normalization converts the summarized flow into several "resolutions." For example, one resolution might be 5000 ms. So the StatisticsManager takes the flow's end time/5000 - start time/5000 and divides the volume by the result and inserts that data into the database. It sounds kind of confusing, but results in a dataset that allows me to query for netflow activity for any time frame.
It also means that the database is queried as it is written to; existing data is updated more frequently than new data is written.
Regardless, Sleepycat seems to be holding on to more data than it needs to. Once an interval has passed (maybe 5 minutes or so), it is unlikely to be accessed again until queried to create charts. However, the memory usage grows out of control despite the lack of necessity for keeping all but recently entered entries in cache.
This is how the trouble generally starts (note that I've increased the timeout to 15000 ms to try to accommodate for longer wait times, but that just seems to delay the onset of the issue by a day or so):
[#|2010-10-23T13:46:00.854-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=64;_ThreadName=Thread-1;|addStatisticalSeconds Failed: (JE 4.0.104) Lock expired. Locker 30264205 -1_pool-6-thread-8_ThreadLocker: waited for lock on database=persist#Statistics#name.justinthomas.flower.analysis.statistics.StatisticalInterval LockAddr:32426808 node=17057517 type=READ grant=WAIT_NEW timeoutMillis=15000 startTime=1287866745448 endTime=1287866760795
Owners: [<LockInfo locker="29616818 -1_pool-6-thread-5_ThreadLocker" type="WRITE"/>]
Waiters: [<LockInfo locker="7246740 -1_pool-6-thread-6_ThreadLocker" type="READ"/>, <LockInfo locker="26940477 -1_pool-6-thread-2_ThreadLocker" type="WRITE"/>, <LockInfo locker="5099094 -1_pool-6-thread-4_ThreadLocker" type="READ"/>]
|#]
[#|2010-10-23T15:00:41.343-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=65;_ThreadName=Thread-1;|addStatisticalSeconds Failed: (JE 4.0.104) Lock expired. Locker 14184769 -1_pool-6-thread-1_ThreadLocker: waited for lock on database=persist#Statistics#name.justinthomas.flower.analysis.statistics.StatisticalInterval LockAddr:32730917 node=17057517 type=READ grant=WAIT_NEW timeoutMillis=15000 startTime=1287871223679 endTime=1287871241341
Owners: [<LockInfo locker="23975039 -1_pool-6-thread-8_ThreadLocker" type="WRITE"/>]
Waiters: [<LockInfo locker="19650664 -1_pool-6-thread-6_ThreadLocker" type="READ"/>]
|#]
[#|2010-10-23T15:32:13.090-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=66;_ThreadName=Thread-1;|addStatisticalSeconds Failed: (JE 4.0.104) Lock expired. Locker 17937364 -1_pool-6-thread-7_ThreadLocker: waited for lock on database=persist#Statistics#name.justinthomas.flower.analysis.statistics.StatisticalInterval LockAddr:20265315 node=17057517 type=WRITE grant=WAIT_NEW timeoutMillis=15000 startTime=1287873113904 endTime=1287873133089
Owners: [<LockInfo locker="3251671 -1_pool-6-thread-1_ThreadLocker" type="WRITE"/>]
Waiters: [<LockInfo locker="32174859 -1_pool-6-thread-8_ThreadLocker" type="READ"/>, <LockInfo locker="33186148 -1_pool-6-thread-4_ThreadLocker" type="WRITE"/>, <LockInfo locker="17825718 -1_pool-6-thread-2_ThreadLocker" type="READ"/>]
|#]
[#|2010-10-23T15:32:13.096-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=64;_ThreadName=Thread-1;|addStatisticalSeconds Failed: (JE 4.0.104) Lock expired. Locker 32174859 -1_pool-6-thread-8_ThreadLocker: waited for lock on database=persist#Statistics#name.justinthomas.flower.analysis.statistics.StatisticalInterval LockAddr:20265315 node=17057517 type=READ grant=WAIT_NEW timeoutMillis=15000 startTime=1287873118064 endTime=1287873133092
Owners: [<LockInfo locker="3251671 -1_pool-6-thread-1_ThreadLocker" type="WRITE"/>]
Waiters: [<LockInfo locker="33186148 -1_pool-6-thread-4_ThreadLocker" type="WRITE"/>, <LockInfo locker="17825718 -1_pool-6-thread-2_ThreadLocker" type="READ"/>]
|#]
[#|2010-10-23T15:32:13.367-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=67;_ThreadName=Thread-1;|addStatisticalSeconds Failed: (JE 4.0.104) Lock expired. Locker 33186148 -1_pool-6-thread-4_ThreadLocker: waited for lock on database=persist#Statistics#name.justinthomas.flower.analysis.statistics.StatisticalInterval LockAddr:20265315 node=17057517 type=WRITE grant=WAIT_NEW timeoutMillis=15000 startTime=1287873118366 endTime=1287873133366
Owners: [<LockInfo locker="3251671 -1_pool-6-thread-1_ThreadLocker" type="WRITE"/>]
Waiters: [<LockInfo locker="17825718 -1_pool-6-thread-2_ThreadLocker" type="READ"/>, <LockInfo locker="25145711 -1_pool-6-thread-6_ThreadLocker" type="READ"/>, <LockInfo locker="5544029 -1_pool-6-thread-5_ThreadLocker" type="READ"/>]
|#]
[#|2010-10-23T15:33:14.030-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=68;_ThreadName=Thread-1;|addStatisticalSeconds Failed: (JE 4.0.104) Lock expired. Locker 31602565 -1_pool-6-thread-5_ThreadLocker: waited for lock on database=persist#Statistics#name.justinthomas.flower.analysis.statistics.StatisticalInterval LockAddr:11219992 node=17057517 type=WRITE grant=WAIT_NEW timeoutMillis=15000 startTime=1287873175916 endTime=1287873194019
Owners: [<LockInfo locker="27649147 -1_pool-6-thread-2_ThreadLocker" type="WRITE"/>]
Waiters: [<LockInfo locker="3895581 -1_pool-6-thread-7_ThreadLocker" type="WRITE"/>, <LockInfo locker="8345933 -1_pool-6-thread-8_ThreadLocker" type="WRITE"/>, <LockInfo locker="12576013 -1_pool-6-thread-6_ThreadLocker" type="WRITE"/>, <LockInfo locker="5695501 -1_pool-6-thread-1_ThreadLocker" type="WRITE"/>]
|#]
[#|2010-10-23T15:33:23.334-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=64;_ThreadName=Thread-1;|addStatisticalSeconds Failed: (JE 4.0.104) Lock expired. Locker 8345933 -1_pool-6-thread-8_ThreadLocker: waited for lock on database=persist#Statistics#name.justinthomas.flower.analysis.statistics.StatisticalInterval LockAddr:11219992 node=17057517 type=WRITE grant=WAIT_NEW timeoutMillis=15000 startTime=1287873184851 endTime=1287873203333
Owners: [<LockInfo locker="3895581 -1_pool-6-thread-7_ThreadLocker" type="WRITE"/>]
Waiters: [<LockInfo locker="12576013 -1_pool-6-thread-6_ThreadLocker" type="WRITE"/>, <LockInfo locker="5695501 -1_pool-6-thread-1_ThreadLocker" type="WRITE"/>, <LockInfo locker="13327115 -1_pool-6-thread-4_ThreadLocker" type="READ"/>, <LockInfo locker="11939897 -1_pool-6-thread-5_ThreadLocker" type="READ"/>]
|#]
[#|2010-10-23T15:33:23.344-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=69;_ThreadName=Thread-1;|addStatisticalSeconds Failed: (JE 4.0.104) Lock expired. Locker 12576013 -1_pool-6-thread-6_ThreadLocker: waited for lock on database=persist#Statistics#name.justinthomas.flower.analysis.statistics.StatisticalInterval LockAddr:11219992 node=17057517 type=WRITE grant=WAIT_NEW timeoutMillis=15000 startTime=1287873184893 endTime=1287873203343
Owners: [<LockInfo locker="3895581 -1_pool-6-thread-7_ThreadLocker" type="WRITE"/>]
Waiters: [<LockInfo locker="5695501 -1_pool-6-thread-1_ThreadLocker" type="WRITE"/>, <LockInfo locker="13327115 -1_pool-6-thread-4_ThreadLocker" type="READ"/>, <LockInfo locker="11939897 -1_pool-6-thread-5_ThreadLocker" type="READ"/>]
|#]
Those errors go on and on and on, until I eventually see this:
[#|2010-10-23T17:37:29.876-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=71;_ThreadName=Thread-1;|Exception in thread "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv]]" |#]
[#|2010-10-23T17:37:34.915-0700|INFO|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=23;_ThreadName=Thread-1;|In main loop, we have serious trouble: java.lang.OutOfMemoryError: Java heap space|#]
[#|2010-10-23T17:37:56.516-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=66;_ThreadName=Thread-1;|Exception in thread "pool-6-thread-7" |#]
[#|2010-10-23T17:39:11.060-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=23;_ThreadName=Thread-1;|Exception in thread "{felix.fileinstall.poll=5000, felix.fileinstall.bundles.new.start=true, service.pid=org.apache.felix.fileinstall.fd8877ce-71aa-41d2-8ddc-15ce996cde1b, felix.fileinstall.dir=/opt/glassfishv3/glassfish/domains/domain1/autodeploy/bundles/, felix.fileinstall.filename=org.apache.felix.fileinstall-autodeploy-bundles.cfg, service.factorypid=org.apache.felix.fileinstall, felix.fileinstall.debug=1}" |#]
[#|2010-10-23T17:39:11.070-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=66;_ThreadName=Thread-1;|java.lang.OutOfMemoryError: Java heap space
at java.util.IdentityHashMap.init(IdentityHashMap.java:261)
at java.util.IdentityHashMap.<init>(IdentityHashMap.java:207)
at com.sleepycat.je.utilint.IdentityHashMap.<init>(IdentityHashMap.java:25)
at com.sleepycat.je.cleaner.LocalUtilizationTracker.<init>(LocalUtilizationTracker.java:39)
at com.sleepycat.je.recovery.Checkpointer.flushDirtyNodes(Checkpointer.java:665)
at com.sleepycat.je.recovery.Checkpointer.syncDatabase(Checkpointer.java:604)
at com.sleepycat.je.dbi.DatabaseImpl.sync(DatabaseImpl.java:977)
at com.sleepycat.je.dbi.DatabaseImpl.handleClosed(DatabaseImpl.java:863)
at com.sleepycat.je.Database.closeInternal(Database.java:458)
at com.sleepycat.je.Database.close(Database.java:314)
at com.sleepycat.je.SecondaryDatabase.close(SecondaryDatabase.java:331)
at com.sleepycat.persist.impl.Store.closeDb(Store.java:1454)
at com.sleepycat.persist.impl.Store.close(Store.java:1059)
at com.sleepycat.persist.EntityStore.close(EntityStore.java:630)
at name.justinthomas.flower.analysis.persistence.FlowReceiver.addFlow(FlowReceiver.java:94)
at name.justinthomas.flower.analysis.persistence.FlowReceiver.addFlow(FlowReceiver.java:65)
at name.justinthomas.flower.collector.FlowWorker.parseData(FlowWorker.java:382)
at name.justinthomas.flower.collector.FlowWorker.v9(FlowWorker.java:111)
at name.justinthomas.flower.collector.FlowWorker.run(FlowWorker.java:61)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
|#]
[#|2010-10-23T17:39:11.124-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=23;_ThreadName=Thread-1;|java.lang.OutOfMemoryError: Java heap space
|#]
[#|2010-10-23T17:39:11.141-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=72;_ThreadName=Thread-1;|Exception in thread "pool-6-thread-3" |#]
[#|2010-10-23T17:39:11.144-0700|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=72;_ThreadName=Thread-1;|java.lang.OutOfMemoryError: Java heap space
|#]
It's very frustrating, because things run so well at first and then just deteriorate into a resource nightmare. Any suggestions would be welcome. The application is running with 3 CPU cores and 2GB RAM.
Edited by: justindthomas on Oct 23, 2010 7:19 PM: I initially tried to use the forum's "URL" mechanism, but that doesn't seem to work worth anything, so I un-did it.I've disabled that thread for now. While debugging that, I ran into a SecondaryIntegrityException and I read that I shouldn't use secondary indexes without also using transactions. I enabled transactional processing, but the locking issues grew far worse. I opted to find ways to not use secondary indexes instead.You're right that with secondaries it is important to use txns. But I'm not sure why you're having such severe locking problems with txns. Were you using a txn with a cursor, to perform a scan? If so, I can probably suggest ways of doing that without the txn, if you can describe what you're doing and/or point me to your source code. Or, perhaps you've decided not to use secondaries, and this isn't an issue anymore?
--mark -
Session Timeouts and SmbServer
Hi,
When having iFS mapped to a network drive (via SMB), the SMB server
is unable to recover from a timeout of the LibrarySession. The network
drive then seems to be empty and doing a refresh within explorer
doesn't help either. The only thing that helps, is remapping the
network drive.
Within Node.log of iFS I see this stacktrace.
7/10/02 9:02 AM SmbServer: oracle.ifs.common.IfsException
oracle.ifs.common.IfsException: IFS-21000: Session is not connected or has timed-out
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Compiled Code)
at java.lang.Throwable.<init>(Compiled Code)
at java.lang.Exception.<init>(Compiled Code)
at oracle.ifs.common.IfsException.<init>(Compiled Code)
at oracle.ifs.common.IfsException.<init>(Compiled Code)
at oracle.ifs.common.IfsException.<init>(Compiled Code)
at oracle.ifs.beans.LibraryObject.verifyConnected(Compiled Code)
at oracle.ifs.beans.Folder.findPublicObjectByPath(Compiled Code)
at oracle.ifs.beans.FolderPathResolver.findPublicObjectByPath(Compiled Code)
at oracle.ifs.beans.FolderPathResolver.findPublicObjectByPath(Compiled Code)
at oracle.ifs.protocols.smb.server.DbTree$DbQuery.<init>(Compiled Code)
at oracle.ifs.protocols.smb.server.DbTree.getQuery(Compiled Code)
at oracle.ifs.protocols.smb.server.ComTrans.trans2FindFirst(Compiled Code)
at oracle.ifs.protocols.smb.server.ComTrans.replyTransaction2(Compiled Code)
at oracle.ifs.protocols.smb.server.ComTrans.process(Compiled Code)
at oracle.ifs.protocols.smb.server.ComSmb.handleSmbMessage(Compiled Code)
at oracle.ifs.protocols.smb.server.SmbThread.handleNbMessage(Compiled Code)
at oracle.ifs.protocols.smb.server.SmbThread.readPackets(Compiled Code)
at oracle.ifs.protocols.smb.server.SmbThread.run(Compiled Code)
This behavior actually causes us big problems when editing files via MS Office.
Fortunately Office is able to still save it's data using some generated filename.
(At least until now I could not create any data loss)
But then you have to close it, remap then network drive, rename the file and then
reopen the file. This is big trouble to users, which are not familiar with mapping
network drives and renaming files with extensions.
Is there a way to make the SmbServer keep the LibrarySession alive, as long as
the network drive is mapped ?
Regards,
Jens LorenzWorkflow #2:
Login to my account
Click view all email
Open Drafts Folder
Open draft email response
Select "Send" to send email (total in session time of 30 seconds)
On screen reload, where I would expect to see some sort of indication that my email was successfully sent, instead the system throws session time out message and kicks me out.
I have no idea if my email was successfully sent or not.
Workflow #3:
Login to my account
Click view all email
Attempted to open the first new email in my inbox (total time in session <15 seconds)
System throws session timeout error and kicks me out to the main login.
There is obviously something going on with your session holding code. The session variable is not being passed correctly or something but it's very, very frustrating to spend 30-45 minutes trying to type out a couple of lines, particularly when you have multiple important activities going on that you need to respond too via email. -
Webmail timeouts and lost mail
I have seen several topics here with people complaining that Webmail times out when they are composing outgoing mail, and lose it when they hit send. However, none of these existing posts seem to have valid fixes or responses, and they all seem to be locked down to prevent new comments, so here is my two cents:
1. I have absolutely experienced the same thing. I've finally gotten into the habit of composing e-mail in an external editor, because I am sick and tired of wasting time recomposing lost e-mail.
2. The Verizon FAQ states two, contradictory policies, both of which are inaccurate:
http://www22.verizon.com/ResidentialHelp/HighSpeed/Email/Setup+And+Use/QuestionsOne/121965.htm:
Does Webmail time me out?
Webmail does not timeout due to inactivity.
http://www22.verizon.com/ResidentialHelp/HighSpeed/Email/Setup%20And%20Use/QuestionsOne/123344.htm
Does Webmail time out?
Due to security reasons, Webmail Basic does timeout after 20 minutes of inactivity. However, messages that were being composed before your session timed out will be saved in the DRAFTS folder.
The first one is outright wrong. Webmail absolutely, positively DOES time out.
The second one is wrong, at least in some circumstances. I have no doubt that this is their desired policy, but like many other customers, I'm here to tell you that under at least some circumstances, Webmai-l times out in as little as 5 minutes, and in these same circumstances, IT DOES NOT SAVE YOUR COMPOSED E-MAIL TO THE DRAFTS FOLDER. It simplay takes you to the log in screen, and when you log back in, your text is irretrievably gone. Verizon Webmail has not auto-saved anything for me in at least a year.
3. I have a theory as to what is going on here, and I am testing it today. Maybe one or two other folks can try this, too. I will report back whatever I find:
I access Webmail through an old bookmark which is defined as http://netmail.verizon.net/webmail/driver?nimlet=deggetfolder
However, I notice that if I login fresh from www.verizon.net and navigate to Webmail, I GET DIFFERENT SESSION COOKIES THAN I DO FROM THE ABOVE BOOKMARK. For example, I get a cookie called -_WL_AUTHCOOKIE_VZCSESSIONID which is not present when I log in using the old bookmark.. There are a couple of other variations, but this seems the most suspicious.
So, my theory is that those of us who often lose composed e-mail after ridiculously short time outs, are logging in from old bookmarks, and that things changed underneath us at some point during a Verizon Webmail update. I am going to try usng Webmail today from a fresh login, after first clearing out all my existing Verizon cookies, and see if this results in longer sessions without timing out, and saved messages when it does time out.Oh, and don't even get me started on SEARCH for text within a message. This has never worked, from Day One.
All I ever get is "We were unable to perform your request. Please try again." SEARCH on header text, like Sender and Subject works fine, but apparently Verizon QA has never tested the other options on large mail inboxes.
I'm pretty forgiving and undersrtanding of minor glitches and shortcomings in software, but Verizon Webmail is one of the most unreliable utilities that I've ever encountered from a large company with a huge customer base. -
NetworkManager timeouts and can't connect
Hello,
I have a fresh install of Arch and I'm not succeeding in connecting with Network Manager. I followed the Wiki article, but everytime I run NetworkManager (either through applet or command line), it fails.
$ nmcli d wifi connect myNetwork
Error: Timeout 90 sec expired
(I get prompted for password a few times)
I checked the general troubleshooting but nothing there seemed to relate to my problem. I'm able to connect with netctl (through wifi-menu), though, which is what I'm using to post here.If you're not using a DE or other packages that require it, you might consider chucking NetworkManager.
I know this isn't really an answer to the question you're asking, but I offer it because I had several networks at my office that I could never connect to with NetworkManager. However, once I switched to connman I haven't had any problems since. Connman doesn't offer an official applet or integration on the level of NetworkManager, but it's much lighter and faster and isn't too hard to configure. The connman wiki article doesn't offer much; I suggest looking at the source's configuration doc for specifics on setting up networks. I'm using qconnman-ui and it seems to work well.
Maybe you are looking for
-
How can I remove an old pc registered to my itunes account/
I am just syncing my ipod with my new laptop and want to transfer purchases to my new laptop but it says that I have too many computers linked to my itunes account. My husband and 2 children have their pcs linked to it and now I want to add my new on
-
Dispute case does not get updated with subsequent partial payment
Hi Experts I am now still testing the Dispute management .. but it's my first time to see that some times created dispute case does not get updated with the payment posted against the invoice which the dispute case has been raised .. when i trying to
-
Hello All, I have a Sony Wireless Headset model CECHYA-0086. It was designed and bought for use with my PS3, however I have been able to use it with windows. I have not had any luck finding driver or product support for Linux and was wondering if any
-
IPad und iPhone auf eine Apple ID aber mit getrennten Nummern
Hallo, mit dem Update auf iOS 6 hat sich ja einiges getan, leider war nicht alles okay. Mein iPad und das iPhone hatten vorher unterschiedliche Nummern (0171 xxxx und 0172 xxxx) unter denen Sie erreichbar waren. Mit dem wechsel des iOS auf 6 wurde da
-
Hi All, I have notice that during the activation of the BC sets, there are 6 langauges (English, German, Portuguese, Japanese, French, Spanish) available in the table content of a transport. I've installed and activated the BC sets in 3 backend syste