Error binding ServerSocket to any of the supplied ports

Question

Hi,
I am using Oracle coherence 3.7 version and below is the my coherence configuration proxy file:
<?xml version="1.0"?>
<cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config coherence-cache-config.xsd">
  <caching-scheme-mapping>
  <cache-mapping>
  <cache-name>*</cache-name>
  <scheme-name>distributedCache</scheme-name>
  </cache-mapping>
  </caching-scheme-mapping>
  <caching-schemes>
  <distributed-scheme>
  <scheme-name>distributedCache</scheme-name>
  <service-name system-property="spreading.ssm.coherence.cacheScheme.serviceName">distributedCacheService</service-name>
  <serializer>
  <instance>
  <class-name>com.rbos.ssm.cache.entity.imp.ExtendedPofContext</class-name>
  <init-params>
  <init-param>
  <param-type>string</param-type>
  <param-value>cluster-coherence-pof-config.xml</param-value>
  </init-param>
  </init-params>
  </instance>
  </serializer>
  <partition-listener>
  <class-name>name of class</class-name>
  </partition-listener>
  <backing-map-scheme>
  <local-scheme>
  <unit-calculator>BINARY</unit-calculator>
  </local-scheme>
  </backing-map-scheme>
  <autostart>true</autostart>
  </distributed-scheme>
  <!-- allows extend clients to connect to this member service-name -->
  <proxy-scheme>
  <scheme-name>extendTcpProxy</scheme-name>
  <service-name>extendTcpProxyService</service-name>
  <thread-count>16</thread-count>
  <acceptor-config>
  <tcp-acceptor>
  <local-address>
  <address>localhost</address>
  <port system-property="port"></port>
  </local-address>
  <reuse-address>true</reuse-address>
  </tcp-acceptor>
  <serializer>
  <instance>
  <class-name>class name</class-name>
  <init-params>
  <init-param>
  <param-type>string</param-type>
  <param-value>cluster-coherence-pof-config.xml</param-value>
  </init-param>
  </init-params>
  </instance>
  </serializer>
  </acceptor-config>
  <load-balancer>client</load-balancer>
  <autostart>true</autostart>
  </proxy-scheme>
  <invocation-scheme>
  <scheme-name>invocationService</scheme-name>
  <service-name system-property="serbice name">invocationService</service-name>
  <autostart>true</autostart>
  </invocation-scheme>
  </caching-schemes>
</cache-config>
I am facing "Error binding ServerSocket to any of the supplied ports" very frequently in my application which in turn causes High CPU Usage issue. Below is the stacktrace of exception:
2014-08-10 15:04:05.035/719374.713 Oracle Coherence GE 3.7.1.4 <D4> (thread=Proxy:extendTcpProxyService:TcpAcceptor:TcpProcessor, member=3): TcpProcessor caught an unhandled exception (java.lang.RuntimeException: Error binding ServerSocket to any of the supplied ports: [trulcch2002/11.179.103.181:7501]) while exiting.
2014-08-10 15:04:05.035/719374.713 Oracle Coherence GE 3.7.1.4 <D4> (thread=Proxy:extendTcpProxyService:TcpAcceptor, member=3): TcpAcceptor caught an unhandled exception (com.tangosol.util.WrapperException: (Wrapped) Error binding ServerSocket to any of the supplied ports: [trulcch2002/11.179.103.181:7501]) while exiting.
2014-08-10 15:04:05.036/719374.714 Oracle Coherence GE 3.7.1.4 <D5> (thread=Proxy:extendTcpProxyService:TcpAcceptor, member=3): Stopped: TcpAcceptor{Name=Proxy:extendTcpProxyService:TcpAcceptor, State=(SERVICE_STOPPED), ThreadCount=0, Codec=Codec(Format=POF), Serializer=com.rbos.ssm.cache.entity.imp.ExtendedPofContext, PingInterval=0, PingTimeout=30000, RequestTimeout=30000, SocketProvider=SystemSocketProvider, LocalAddress=[trulcch2002/11.179.103.181:7501], SocketOptions{LingerTimeout=0, KeepAliveEnabled=true, TcpDelayEnabled=false}, ListenBacklog=0, BufferPoolIn=BufferPool(BufferSize=2KB, BufferType=DIRECT, Capacity=Unlimited), BufferPoolOut=BufferPool(BufferSize=2KB, BufferType=DIRECT, Capacity=Unlimited)}
2014-08-10 15:04:05.037/719374.715 Oracle Coherence GE 3.7.1.4 <D4> (thread=Proxy:extendTcpProxyService, member=3): ProxyService caught an unhandled exception (com.tangosol.util.WrapperException: (Wrapped) (Wrapped) Error binding ServerSocket to any of the supplied ports: [trulcch2002/11.179.103.181:7501]) while exiting.
2014-08-10 15:04:05.037/719374.715 Oracle Coherence GE 3.7.1.4 <Error> (thread=Proxy:extendTcpProxyService, member=3): ClusterService.doServiceLeft: Unknown Service ProxyService{Name=extendTcpProxyService, State=(SERVICE_STOPPED), Id=4, Version=3.7}
2014-08-10 15:04:05.037/719374.716 Oracle Coherence GE 3.7.1.4 <D5> (thread=Proxy:extendTcpProxyService, member=3): Service extendTcpProxyService left the cluster
2014-08-10 15:04:05.039/719374.717 Oracle Coherence GE 3.7.1.4 <Error> (thread=main, member=3): Error while starting service "extendTcpProxyService": (Wrapped) (Wrapped) (Wrapped) java.lang.RuntimeException: Error binding ServerSocket to any of the supplied ports: [trulcch2002/11.179.103.181:7501]
  at com.tangosol.coherence.component.util.Daemon.start(Daemon.CDB:52)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:7)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
  at com.tangosol.coherence.component.util.SafeService.startService(SafeService.CDB:39)
  at com.tangosol.coherence.component.util.SafeService.restartService(SafeService.CDB:28)
  at com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:39)
  at com.tangosol.coherence.component.util.SafeService.start(SafeService.CDB:14)
  at com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInternal(DefaultConfigurableCacheFactory.java:1105)
  at com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:937)
  at com.tangosol.net.DefaultCacheServer.startServices(DefaultCacheServer.java:81)
  at com.tangosol.net.DefaultCacheServer.monitorServices(DefaultCacheServer.java:285)
  at com.tangosol.net.DefaultCacheServer.startAndMonitor(DefaultCacheServer.java:56)
  at com.tangosol.net.DefaultCacheServer.main(DefaultCacheServer.java:197)
Caused by: (Wrapped) (Wrapped) java.lang.RuntimeException: Error binding ServerSocket to any of the supplied ports: [trulcch2002/11.179.103.181:7501]
  at com.tangosol.coherence.component.util.Daemon.start(Daemon.CDB:52)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:7)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ProxyService.onServiceStarting(ProxyService.CDB:37)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onServiceState(Grid.CDB:12)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.setServiceState(Service.CDB:8)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.setServiceState(Grid.CDB:21)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onEnter(Grid.CDB:32)
  at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:14)
  at java.lang.Thread.run(Unknown Source)
Caused by: (Wrapped) java.lang.RuntimeException: Error binding ServerSocket to any of the supplied ports: [trulcch2002/11.179.103.181:7501]
  at com.tangosol.coherence.component.util.Daemon.start(Daemon.CDB:52)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor.onServiceStarting(TcpAcceptor.CDB:7)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onServiceState(Service.CDB:4)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.setServiceState(Service.CDB:8)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.onEnter(Peer.CDB:11)
  ... 2 more
Caused by: java.lang.RuntimeException: Error binding ServerSocket to any of the supplied ports: [trulcch2002/11.179.103.181:7501]
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor.configureSocket(TcpAcceptor.CDB:39)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor$TcpProcessor.onEnter(TcpAcceptor.CDB:26)
  ... 2 more
2014-08-10 15:04:05.039/719374.717 Oracle Coherence GE 3.7.1.4 <Error> (thread=main, member=3): Failed to restart services: (Wrapped) (Wrapped) (Wrapped) java.lang.RuntimeException: Error binding ServerSocket to any of the supplied ports: [trulcch2002/11.179.103.181:7501]
2014-08-10 15:04:10.041/719379.719 Oracle Coherence GE 3.7.1.4 <Info> (thread=main, member=3): Restarting Service: extendTcpProxyService
2014-08-10 15:04:10.047/719379.725 Oracle Coherence GE 3.7.1.4 <Error> (thread=Proxy:extendTcpProxyService:TcpAcceptor:TcpProcessor, member=3): error binding ServerSocket to 11.179.103.181:7501: java.net.BindException: Address already in use
  at sun.nio.ch.Net.bind(Native Method)
  at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
  at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor.configureSocket(TcpAcceptor.CDB:25)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor$TcpProcessor.onEnter(TcpAcceptor.CDB:26)
  at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:14)
  at java.lang.Thread.run(Unknown Source)
Please help me in this issue. Thanks in advance.

Answer 1

Hi,
I've looked at the code where the socket is created. Unfortunately, the ServerSocket constructor does not take the options as a parameter, so it is not a configuration option with Coherence. However, we would be able to support SO_REUSEADDR by using the default constructor, setting the desired options, and deferring the bind until that has been done.
I've assigned this the tracking number COH-2724. If you'd be willing to test a patch for it, send me your email address (my first name dot my last name at oracle dot com) and I will log that information onto the issue.
Peace,
Cameron Purdy | Oracle Coherence
http://coherence.oracle.com/