Cluster is getting restarted after IndexOutOfBoundsException

Question

Hi,
While working with the Coherence push replication pattern (Coherence Incubator - Release 10) , I am getting this below exception on active node and the cluster is getting restarted, so the cache objects will be lost.
Any help on this exception ?
Thanks in advance.
++++++++++++++ Exception stack trace +++++++++++++++++++++++++++++
04-16-2013 01:33:13 [ERROR] Logger@9251739 3.7.1.0 (Log4j.CDB:3) - 2013-04-16 01:33:13.679/3392.475 Oracle Coherence GE 3.7.1.0 <Error> (thread=PacketPublisher, member=1): Stopping cluster due to unhandled exception: java.lang.IndexOutOfBoundsException: of=23, cb=-1962431982, max=2147483647
at com.tangosol.io.ByteArrayWriteBuffer.boundsException(ByteArrayWriteBuffer.java:452)
at com.tangosol.io.ByteArrayWriteBuffer.checkBounds(ByteArrayWriteBuffer.java:430)
at com.tangosol.io.ByteArrayWriteBuffer.write(ByteArrayWriteBuffer.java:134)
at com.tangosol.io.AbstractWriteBuffer.write(AbstractWriteBuffer.java:50)
at com.tangosol.io.AbstractWriteBuffer$AbstractBufferOutput.writeBuffer(AbstractWriteBuffer.java:867)
at com.tangosol.coherence.component.net.packet.MessagePacket.writeBody(MessagePacket.CDB:8)
at com.tangosol.coherence.component.net.packet.messagePacket.Directed.write(Directed.CDB:42)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onPacket(PacketPublisher.CDB:130)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onNotify(PacketPublisher.CDB:44)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:662)
04-16-2013 01:33:13 [ERROR] Logger@9251739 3.7.1.0 (Log4j.CDB:3) - 2013-04-16 01:33:13.910/3392.706 Oracle Coherence GE 3.7.1.0 <Error> (thread=EventChannelController:Thread-3, member=n/a): EventChannelController EventChannelController.Identifier{symbolicName=Active Publisher, externalName=<undefined>:BermudaCacheCluster_Prod:DistrubutedLPACache:Active Publisher} failed to distribute events Class:com.oracle.coherence.patterns.eventdistribution.distributors.AbstractEventChannelController Method:onDistribute
04-16-2013 01:33:13 [ INFO] Logger@9251739 3.7.1.0 (Log4j.CDB:3) - 2013-04-16 01:33:13.910/3392.706 Oracle Coherence GE 3.7.1.0 <Info> (thread=EventChannelController:Thread-3, member=n/a): Exception was as follows: Class:com.oracle.coherence.patterns.eventdistribution.distributors.AbstractEventChannelController Method:onDistribute
com.tangosol.net.RequestPolicyException: Service has been terminated
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.onMissingStorage(PartitionedCache.CDB:32)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.ensureRequestTarget(PartitionedCache.CDB:22)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.sendPartitionedRequest(PartitionedCache.CDB:24)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.getAll(PartitionedCache.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.getAll(PartitionedCache.CDB:16)
at com.tangosol.util.ConverterCollections$ConverterCacheMap.getAll(ConverterCollections.java:2603)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.getAll(PartitionedCache.CDB:5)
at com.tangosol.coherence.component.util.SafeNamedCache.getAll(SafeNamedCache.CDB:1)
at com.oracle.coherence.patterns.eventdistribution.distributors.coherence.CoherenceEventChannelController.getEventsToDistribute(CoherenceEventChannelController.java:254)
at com.oracle.coherence.patterns.eventdistribution.distributors.AbstractEventChannelController.onDistribute(AbstractEventChannelController.java:892)
at com.oracle.coherence.patterns.eventdistribution.distributors.AbstractEventChannelController.access$200(AbstractEventChannelController.java:70)
at com.oracle.coherence.patterns.eventdistribution.distributors.AbstractEventChannelController$10.run(AbstractEventChannelController.java:993)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
4-16-2013 01:33:16 [ INFO] Logger@9251739 3.7.1.0 (Log4j.CDB:3) - 2013-04-16 01:33:16.745/3395.541 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Restarting cluster
04-16-2013 01:33:16 [ WARN] Logger@9251739 3.7.1.0 (Log4j.CDB:3) - 2013-04-16 01:33:16.746/3395.542 Oracle Coherence GE 3.7.1.0 <Warning> (thread=main, member=n/a): UnicastUdpSocket failed to set receive buffer size to 32 packets (1.99MB); actual size is 6%, 2 packets (127KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.
04-16-2013 01:33:16 [ WARN] Logger@9251739 3.7.1.0 (Log4j.CDB:3) - 2013-04-16 01:33:16.747/3395.543 Oracle Coherence GE 3.7.1.0 <Warning> (thread=main, member=n/a): PreferredUnicastUdpSocket failed to set receive buffer size to 468 packets (3.99MB); actual size is 3%, 14 packets (127KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.
+++++++++++++ Cache config file ++++++++++++++++++++
<caching-scheme-mapping>
<cache-mapping>
<cache-name>DistrubutedLPACache</cache-name>
<scheme-name>LPANewCache</scheme-name>
<!-- Begin Event distrubution config -->
<event:distributor>
<event:distributor-name>DistrubutedLPACache</event:distributor-name>
<event:distributor-external-name>site-BermudaCacheCluster_Prod-DistrubutedLPACache</event:distributor-external-name>
<event:distributor-scheme>
<event:coherence-based-distributor-scheme />
</event:distributor-scheme>
<event:distribution-channels>
<event:distribution-channel>
<event:channel-name>Active Publisher</event:channel-name>
<event:starting-mode system-property="channel.starting.mode">enabled</event:starting-mode>
<event:channel-scheme>
<event:remote-cluster-channel-scheme>
<event:remote-invocation-service-name>remote-site2</event:remote-invocation-service-name>
<event:remote-channel-scheme>
<event:local-cache-channel-scheme>
<event:target-cache-name>DistrubutedLPACache</event:target-cache-name>
</event:local-cache-channel-scheme>
</event:remote-channel-scheme>
</event:remote-cluster-channel-scheme>
</event:channel-scheme>
</event:distribution-channel>
</event:distribution-channels>
</event:distributor>
<!-- End Event distrubution config-->
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<!-- Distributed caching scheme. -->
<distributed-scheme>
<scheme-name>LPANewCache</scheme-name>
<service-name>HBaseLPACache</service-name>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<class-scheme>
<class-name>com.tangosol.util.ObservableHashMap</class-name>
</class-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com.****.pushreplication.PublishingCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>DistrubutedLPACache</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<read-only>false</read-only>
<write-delay-seconds>0</write-delay-seconds>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
<proxy-scheme>
<service-name>BermudaTCPProxyService</service-name>
<thread-count>5</thread-count>
<acceptor-config>
<tcp-acceptor>
<local-address>
<address>localhost</address>
<port system-property="tangosol.coherence.extend.port">9098</port>
</local-address>
</tcp-acceptor>
</acceptor-config>
<autostart>true</autostart>
</proxy-scheme>
<remote-invocation-scheme>
<service-name>remote-site2</service-name>
<initiator-config>
<tcp-initiator>
<remote-addresses>
<socket-address>
<!-- remote cluster server name along with TCP port -->
<address>C605PGTCTASPE</address>
<port>20002</port>
</socket-address>
</remote-addresses>
<connect-timeout>5s</connect-timeout>
</tcp-initiator>
<outgoing-message-handler>
<request-timeout>5s</request-timeout>
</outgoing-message-handler>
</initiator-config>
</remote-invocation-scheme>
</caching-schemes>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Edited by: 966505 on Apr 16, 2013 2:47 AM

Answer 1

Hi Ashwin
It might be dut to JVM-out of memory error. They might have done the settings in the JVM i.e it will restart the JVm when out of memory. So please check this
Also
By default AUTO_START of the Service in JDBC server is set to restore, for a automatic restart it should be set to 1.
So Please check this.
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/f0820f18-7ed6-2910-1d81-9c5929352514
Regards
Abhishek Mahajan
**Reward points if helpful**