TCP Extend (DefaultCacheServer rejects connections)

Hi guys
Have been trying to setup TCP Extend to make a Linux box use cache configured on a windows box and the DefaultCacheServer rejects TCP connections. The config files I'm using are attached. Can anyone help ?
The DefaultCacheServer comes up nicely
SafeCluster: Name=n/a
Group{Address=224.3.2.0, Port=32367, TTL=1}
MasterMemberSet
ThisMember=Member(Id=1, Timestamp=2007-03-29 16:07:16.026, Address=147.114.162.160:54321, MachineId=17312)
OldestMember=Member(Id=1, Timestamp=2007-03-29 16:07:16.026, Address=147.114.162.160:54321, MachineId=17312)
ActualMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2007-03-29 16:07:16.026, Address=147.114.162.160:54321, MachineId=17312)
RecycleMillis=120000
RecycleSet=MemberSet(Size=0, BitSetCount=0
Services
TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=147.114.162.160:54321}, Connections=[]}
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.2, OldestMemberId=1}
DistributedCache{Name=DistributedCache, State=(SERVICE_STARTED), Id=1, Version=3.2, OldestMemberId=1, LocalStorage=enabled, PartitionCount=257, Bac
upCount=1, AssignedPartitions=257, BackupPartitions=0}
but when I run the client, I get this
2007-03-29 16:09:42.698 Tangosol Coherence DGE 3.2/367 <D4> (thread=TcpRingListener, member=1): Rejecting connection to member 649 using TcpSocket{Sta
te=STATE_OPEN, Socket=Socket[addr=/172.26.102.115,port=36952,localport=54321]} Attachment: cluster-side-config.xml (*To use this attachment you will need to rename 516.bin to cluster-side-config.xml after the download is complete.) Attachment: client-side-config.xml (*To use this attachment you will need to rename 517.bin to client-side-config.xml after the download is complete.)

Hi pandeyv,
You need to configure an instance of the ProxyService in your cluster-side cache configuration file. Coherence*Extend clients connect to the ProxyService over TCP/IP and not the TcpRingService. The TcpRingService is only used by cluster members for death detection.
See the following for instructions on configuring the cluster and client-side configuration files:
http://wiki.tangosol.com/display/COH32UG/Configuring+and+Using+Coherence*Extend
Additionally, I noticed that you are using an old release of Coherence 3.2. Please upgrade to the latest 3.2 service pack (3.2.2):
http://www.tangosol.com/product-downloads.jsp
Regards,
Jason

Similar Messages

Cache config for distributed cache and TCP*Extend

Hi,
I want to use distributed cache with TCP*Extend. We have defined "remote-cache-scheme" as the default cache scheme. I want to use a distributed cache along with a cache-store. The configuration I used for my scheme was
<distributed-scheme>
<scheme-name>MyScheme</scheme-name>
<backing-map-scheme>
 <read-write-backing-map-scheme>
 <internal-cache-scheme>
 <class-scheme>
 <class-name>com.tangosol.util.ObservableHashMap</class-name>
 </class-scheme>
 </internal-cache-scheme>
 <cachestore-scheme>
 <class-scheme>
 <class-name>MyCacheStore</class-name>
 </class-scheme>
 <remote-cache-scheme>
 <scheme-ref>default-scheme</scheme-ref>
 </remote-cache-scheme>
 </cachestore-scheme>
 <rollback-cachestore-failures>true</rollback-cachestore-failures>
 </read-write-backing-map-scheme>
</backing-map-scheme>
</distributed-scheme>
<remote-cache-scheme>
<scheme-name>default-scheme</scheme-name>
<initiator-config>
 <tcp-initiator>
 <remote-addresses>
 <socket-address>
 <address>XYZ</address>
 <port>9909</port>
 </socket-address>
 </remote-addresses>
 </tcp-initiator>
</initiator-config>
</remote-cache-scheme>I know that the configuration defined for "MyScheme" is wrong but I do not know how to configure "MyScheme" correctly to make my distributed cache the part of the same cluster to which all other caches, which uses the default scheme, are joined. Currently, this ain't happening.
Thanks.
RG
Message was edited by:
user602943

Hi,
Is it that I need to define my distributed scheme with the CacheStore in the server-coherence-cache-config.xml and then on the client side use remote cache scheme to connect to get my distributed cache?
Thanks,

OutOfMemoryError in TCP Extend nodes?

Hi,
We are facing a starnge issue in which a storage disabled tcp extend node is consistently occupying around 450mb of space?
Whenever we populate data in cache even smaller than 40mb the it starts filling up oblivious space in tcp extend nodes. Although the jmx console doesn't provide any details of data residing in the storage disabled extend nodes.
Ideally how much memory does the extend node storage disabled tcp extend node requires to run? Is it propotional to max amount of data loaded as a single unit. I feel the extend node should hold data for short duration and should free itself.
We are running on DGE cluster with 2 linux machines having 11 cache nodes of 1gb each. Each machine has 2 storage disabled Extend nodes with 500m each through which my client connects.
Thanks
-Amit
Message was edited by:
Rock Solid

Is there any standard way of deciding on the space required by Extend TCP nodes (Storage Disabled)?
In my scnario
We are trying to get a Map Object of 42mb from the cache. The TCP extend node to which the client connects is throwing OutOfMemoryError. We are runing 4 Extend Nodes of 600MB each.
By looking at the JMX console AvailableMB is < 200 MB consistently in Extend TCP Node (Storage disabled).
Why is this Memory not released after the data transfer is done?
Thanks
-Amit

Lock Client-Side TCP-Extends

Hi,
I am running 2 processes on the same server both connecting to one cluster node.
I need to ensure only one process performs a write operation @ a certain time (i.e. the first one to attain the lock).
I am finding that both the processes are getting the lock, and cannot work-out why...
We are using tcp-extends, and I am using the following related cluster configuration:
<caching-scheme-mapping>
<cache-mapping>
<cache-name>cache.cluster.*</cache-name>
<scheme-name>scheme.cluster.system</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<proxy-scheme>
<scheme-name>scheme.cluster.proxy</scheme-name>
<service-name>service.cluster.proxy</service-name>
<thread-count>4</thread-count>
<acceptor-config>
<tcp-acceptor>
<local-address>
<address system-property="datacloud.node.tcp-extend.bind-address">localhost</address>
<port system-property="datacloud.node.tcp-extend.port">11200</port>
</local-address>
<keep-alive-enabled>true</keep-alive-enabled>
</tcp-acceptor>
</acceptor-config>
<proxy-config>
<cache-service-proxy>
<lock-enabled>true</lock-enabled>

</cache-service-proxy>
</proxy-config>
<autostart>true</autostart>
</proxy-scheme>
<replicated-scheme>
<scheme-name>scheme.cluster.system</scheme-name>
<service-name>ReplicatedCache</service-name>
<lease-granularity>member</lease-granularity>
<member-listener>
<class-name>datacloud.cluster.listeners.ClusterMemberListener</class-name>
</member-listener>
<backing-map-scheme>
<local-scheme />
</backing-map-scheme>
<autostart>true</autostart>
</replicated-scheme>
</caching-schemes>
The client copies this scheme by using the following:
 <cache-mapping>
 <cache-name>cache.cluster.lock</cache-name>
 <scheme-name>scheme.remote</scheme-name>
 </cache-mapping>
The lock essentially does:
NamedCache cache = CacheFactory.getCache("cache.cluster.lock")
boolean isLockAcquired = cache.lock("KEY", 5000)
try {
if(isLockAcquired) {
(takes 2 seconds to complete)
} finally {
cache.unlock("KEY")
Why can 2 processes acquire the same lock when asked to acquire it @ the same time?
Edited by: 907011 on 10-Jan-2012 03:33

Hi,
This is my explination of why you are seeing the behaviour you are - I think it is correct but I am sure someone will jump in if not.
1. Your cache config has <lease-granularity>member</lease-granularity> which means that a lock taken out by any thread on a member can be released by the same member. It also means that a Member owns the lock so if I do cache.lock() for a key from some code running on a Member and then do cache.lock() again for the same key on the same Member they will both succeed as the Member owns the lock.
2. Now, when your first process calls <tt>boolean isLockAcquired = cache.lock("KEY", 5000)</tt> it gets the lock but it is not your client process that owns the lock, it is the Extend Proxy your client is connected to that owns it.
3. Consequently when process two, connected to the same Extend proxy, asks for the lock, it gets it too, as it is not your client process that owns the lock, it is the Extend Proxy your client is connected to that owns it.
4. Worse, when process one finishes and releases the lock, then it is released, even though your code in process two still thinks it is running inside the lock.
As I said, that is my understanding of it and why locks do not really work from Extend Clients. In fact there are very few occasions where I would bother to use explicit locks in Coherence (hence my unfamiliarity about the exact workings) as there are usually other, more reliable, ways to achieve the same requirements.
JK

TCP* Extend client thread pool

Hi,
Is there a way to configure the number of threads used by TCP* Extend client? What is the default value for the same?
For some reason I am observing TCP connection being reset. Here are the logs:
2010-05-05 04:39:02.572/15821.6> (thread=DistributedCacheForHDElements-NY:TcpInitiator, member=n/a): Closed: TcpConnection(Id=0x0000012866AD9AC4AAF0E60D08143308BF16B5D3C3356683C13C90CC0213FB3C, Open=false, LocalAddress=170.240.228.192:1105, RemoteAddress=170.240.230.13:27001)
2010-05-05 04:39:02.572/15821.5> (thread=DistributedCacheForHDElements-NY:TcpInitiator, member=n/a): Stopped: TcpInitiator{Name=DistributedCacheForHDElements-NY:TcpInitiator, State=(SERVICE_STOPPED), ThreadCount=0, Codec=Codec(Format=POF), PingInterval=0, PingTimeout=10000, RequestTimeout=10000, ConnectTimeout=10000, RemoteAddresses=[/170.240.230.13:27001,/141.128.62.137:27007,/141.128.62.138:27005,/170.240.230.13:27002,/141.128.62.138:27006,/141.128.62.137:27008,/170.240.230.13:27003,/141.128.62.138:27004], KeepAliveEnabled=true, TcpDelayEnabled=false, ReceiveBufferSize=0, SendBufferSize=0, LingerTimeout=-1}
2010-05-05 04:39:02.588/15821.5> (thread=DistributedCacheForHDElements-NY:TcpInitiator, member=n/a): Started: TcpInitiator{Name=DistributedCacheForHDElements-NY:TcpInitiator, State=(SERVICE_STARTED), ThreadCount=0, Codec=Codec(Format=POF), PingInterval=0, PingTimeout=10000, RequestTimeout=10000, ConnectTimeout=10000, RemoteAddresses=[/141.128.62.138:27006,/141.128.62.138:27005,/170.240.230.13:27003,/141.128.62.138:27004,/170.240.230.13:27002,/141.128.62.137:27007,/141.128.62.137:27008,/170.240.230.13:27001], KeepAliveEnabled=true, TcpDelayEnabled=false, ReceiveBufferSize=0, SendBufferSize=0, LingerTimeout=-1}
2010-05-05 04:39:02.588/15821.5> (thread=[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Opening Socket connection to 141.128.62.138:27006
2010-05-05 04:39:02.588/15821.nfo> (thread=[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Connected to 141.128.62.138:27006
2010-05-05 04:39:02.604/15821.6> (thread=DistributedCacheForHDElements-NY:TcpInitiator, member=n/a): Opened: TcpConnection(Id=0x00000128679E39D68D803E8A52F48499974C7DC9B4BF127ECF23CF2771B8CB90, Open=true, LocalAddress=170.240.228.192:4742, RemoteAddress=141.128.62.138:27006)
2010-05-05 04:39:02.604/15821.6> (thread=DistributedCacheForHDElements-NY:TcpInitiator, member=n/a): Opened: Channel(Id=1628408480, Open=true, Connection=0x00000128679E39D68D803E8A52F48499974C7DC9B4BF127ECF23CF2771B8CB90)
2010-05-05 04:39:02.619/15821.6> (thread=DistributedCacheForHDElements-NY:TcpInitiator, member=n/a): Opened: Channel(Id=734361514, Open=true, Connection=0x00000128679E39D68D803E8A52F48499974C7DC9B4BF127ECF23CF2771B8CB90)
Regards,
Kishore
Edited by: user10737736 on May 6, 2010 5:06 AM

Hi Kishore,
In the proxy-scheme of your cache configuration, you can use thread-count to configure the number of threads used by TCP* Extend client. The default value is 0. e.g.
<proxy-scheme>
<service-name>ExtendTcpProxyService</service-name>
<thread-count>50</thread-count>
</proxy-scheme>
-Luk

TCP Extend Server - Failed to start Service - Oracle Coherence GE 3.5.2/463

Hello,
We are about to go to production I see Failed to start Service in TCP Extend Server (Storage disable node).
Regards
/AG
My Configuration look like the following
<?xml version="1.0" encoding="windows-1252" ?>
<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config>
<caching-scheme-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>distributedCache</scheme-name>
<service-name>distributedCache</service-name>
<backing-map-scheme>
<local-scheme>
<eviction-policy>HYBRID</eviction-policy>
<high-units>500</high-units>
<low-units>375</low-units>
<unit-calculator>BINARY</unit-calculator>
<unit-factor>1048576</unit-factor>
</local-scheme>
</backing-map-scheme>
</distributed-scheme>
<proxy-scheme>
<service-name>ExtendTcpProxyService</service-name>
<thread-count>15</thread-count>
<acceptor-config>
<tcp-acceptor>
<local-address>
<address system-property="proxy.listen.address">....</address>
<port system-property="proxy.listen.port">....</port>
</local-address>
</tcp-acceptor>
</acceptor-config>
<autostart>true</autostart>
</proxy-scheme>
</caching-schemes>
</cache-config>
------------------------------------------------------------------------- And the Log looks like the following ------------------------------------
2009-12-04 16:21:54.056/25821.278 Oracle Coherence GE 3.5.2/463 <D6> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=12): Closed: Channel(Id=193159068
6, Open=false)
2009-12-04 16:21:54.058/25821.280 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:4, member=12): Repeating SizeReques
t due to the re-distribution of PartitionSet{220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242
, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256}
2009-12-04 16:21:54.058/25821.280 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:12, member=12): Repeating SizeReque
st due to the re-distribution of PartitionSet{220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 24
2, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256}
2009-12-04 16:21:54.058/25821.280 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:11, member=12): Repeating SizeReque
st due to the re-distribution of PartitionSet{220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 24
2, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256}
2009-12-04 16:21:54.058/25821.280 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:7, member=12): Repeating SizeReques
t due to the re-distribution of PartitionSet{220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242
, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256}
2009-12-04 16:21:54.175/25821.397 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:10, member=12): An exception occurr
ed while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:107)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureRequestTarget(DistributedCache.CDB:15)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.sendPartitionedRequest(DistributedCache.CDB:31)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.size(DistributedCache.CDB:13)
at com.tangosol.util.ConverterCollections$ConverterMap.size(ConverterCollections.java:1470)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.size(DistributedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.size(SafeNamedCache.CDB:1)
at com.tangosol.coherence.component.util.collections.WrapperMap.size(WrapperMap.CDB:1)
at com.tangosol.coherence.component.net.extend.messageFactory.NamedCacheFactory$SizeRequest.onRun(NamedCacheFactory.CDB:7)
at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.onMessage(NamedCacheProxy.CDB:11)
at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:28)
at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer$DaemonPool$WrapperTask.run(Peer.CDB:9)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:96)
... 18 more
2009-12-04 16:21:54.175/25821.397 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:9, member=12): An exception occurre
d while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:107)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureRequestTarget(DistributedCache.CDB:15)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.sendPartitionedRequest(DistributedCache.CDB:31)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.size(DistributedCache.CDB:13)
at com.tangosol.util.ConverterCollections$ConverterMap.size(ConverterCollections.java:1470)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.size(DistributedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.size(SafeNamedCache.CDB:1)
at com.tangosol.coherence.component.util.collections.WrapperMap.size(WrapperMap.CDB:1)
at com.tangosol.coherence.component.net.extend.messageFactory.NamedCacheFactory$SizeRequest.onRun(NamedCacheFactory.CDB:7)
at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.onMessage(NamedCacheProxy.CDB:11)
at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:28)
at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer$DaemonPool$WrapperTask.run(Peer.CDB:9)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:96)
... 18 more
2009-12-04 16:21:54.175/25821.397 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:13, member=12): An exception occurr
ed while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:107)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureRequestTarget(DistributedCache.CDB:15)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.sendPartitionedRequest(DistributedCache.CDB:31)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.size(DistributedCache.CDB:13)
at com.tangosol.util.ConverterCollections$ConverterMap.size(ConverterCollections.java:1470)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.size(DistributedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.size(SafeNamedCache.CDB:1)
at com.tangosol.coherence.component.util.collections.WrapperMap.size(WrapperMap.CDB:1)
at com.tangosol.coherence.component.net.extend.messageFactory.NamedCacheFactory$SizeRequest.onRun(NamedCacheFactory.CDB:7)
at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.onMessage(NamedCacheProxy.CDB:11)
at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:28)
at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer$DaemonPool$WrapperTask.run(Peer.CDB:9)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:96)
... 18 more
2009-12-04 16:21:54.175/25821.397 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:11, member=12): An exception occurr
ed while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:107)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureRequestTarget(DistributedCache.CDB:15)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.sendPartitionedRequest(DistributedCache.CDB:31)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.size(DistributedCache.CDB:13)
at com.tangosol.util.ConverterCollections$ConverterMap.size(ConverterCollections.java:1470)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.size(DistributedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.size(SafeNamedCache.CDB:1)
at com.tangosol.coherence.component.util.collections.WrapperMap.size(WrapperMap.CDB:1)
at com.tangosol.coherence.component.net.extend.messageFactory.NamedCacheFactory$SizeRequest.onRun(NamedCacheFactory.CDB:7)
at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.onMessage(NamedCacheProxy.CDB:11)
at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:28)
at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer$DaemonPool$WrapperTask.run(Peer.CDB:9)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:96)
... 18 more
2009-12-04 16:21:54.175/25821.397 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:2, member=12): An exception occurre
d while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:107)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureRequestTarget(DistributedCache.CDB:15)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.sendPartitionedRequest(DistributedCache.CDB:31)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.size(DistributedCache.CDB:13)
at com.tangosol.util.ConverterCollections$ConverterMap.size(ConverterCollections.java:1470)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.size(DistributedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.size(SafeNamedCache.CDB:1)
at com.tangosol.coherence.component.util.collections.WrapperMap.size(WrapperMap.CDB:1)
at com.tangosol.coherence.component.net.extend.messageFactory.NamedCacheFactory$SizeRequest.onRun(NamedCacheFactory.CDB:7)
at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.onMessage(NamedCacheProxy.CDB:11)
at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:28)
at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer$DaemonPool$WrapperTask.run(Peer.CDB:9)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:96)
... 18 more
2009-12-04 16:21:54.176/25821.398 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:6, member=12): An exception occurre
d while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
.CDB:107)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
:$
2009-12-04 16:21:54.259/25821.481 Oracle Coherence GE 3.5.2/463 <D4> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:8, member=12): Daemon caught an unh
andled exception (com.tangosol.net.messaging.ConnectionException: channel is closed) while exiting.
2009-12-04 16:21:54.264/25821.486 Oracle Coherence GE 3.5.2/463 <D4> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:3, member=12): Daemon caught an unh
andled exception (com.tangosol.net.messaging.ConnectionException: channel is closed) while exiting.
2009-12-04 16:21:54.330/25821.552 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=12): Stopped: TcpAcceptor{Name=Pr
oxy:ExtendTcpProxyService:TcpAcceptor, State=(SERVICE_STOPPED), ThreadCount=0, Codec=Codec(Format=POF), PingInterval=0, PingTimeout=0, RequestTimeout=0, Local
Address=[nybc94lxb01/10.12.101.81:21005], LocalAddressReusable=false, KeepAliveEnabled=true, TcpDelayEnabled=false, ReceiveBufferSize=0, SendBufferSize=0, Lis
tenBacklog=0, LingerTimeout=-1, BufferPoolIn=BufferPool(BufferSize=2KB, BufferType=DIRECT, Capacity=Unlimited), BufferPoolOut=BufferPool(BufferSize=2KB, Buffe
rType=DIRECT, Capacity=Unlimited)}
Exception in thread "Thread-2" java.lang.RuntimeException: Failed to start Service "Proxy:ExtendTcpProxyService:TcpAcceptor" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:10)
at com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.closeChannel(Peer.CDB:18)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.closeChannel(Peer.CDB:1)
at com.tangosol.coherence.component.net.extend.Channel.close(Channel.CDB:20)
at com.tangosol.coherence.component.net.extend.Channel.close(Channel.CDB:1)
at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.run(NamedCacheProxy.CDB:30)
at java.lang.Thread.run(Thread.java:619)
Exception in thread "Thread-3" java.lang.RuntimeException: Failed to start Service "Proxy:ExtendTcpProxyService:TcpAcceptor" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:10)
at com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.closeChannel(Peer.CDB:18)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.closeChannel(Peer.CDB:1)
at com.tangosol.coherence.component.net.extend.Channel.close(Channel.CDB:20)
at com.tangosol.coherence.component.net.extend.Channel.close(Channel.CDB:1)
at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.run(NamedCacheProxy.CDB:30)
at java.lang.Thread.run(Thread.java:619)
Edited by: Anand Gupta on Dec 4, 2009 5:10 PM

David,
Thanks for your detailed response. I will try all that you suggested and let you know my observation at the end. To give a back ground of the application I am working on.
Background
Try to replace home grown cache (very well optimized uses custom serialization using NIO bytebuffer etc.,) my mandate is to replace the legacy cache with coherence (I love coherence product (at least 3.4) from my previous job experience). Since all kind of optimization is done in legacy application I am converting NIO byteBuffer generated by legacy serialization to byteArray and converting ByteArray into POF. Also for various other reasons, I have to send entry processor to do even get and put data from the cluster.
Raw Data Set Size
My data size is just 500MB (In Production). I put enough cache nodes so that raw data is just 50MB per node. (Since it is compute heavy with entry processor and the number of clients are going to be ~2500)
What was happening when I got the exception?
I was doing stress testing of the cache. Not sure if that put lot of garbage in some of the node hence may be the garbage collection pause triggered the re-distribution of partition set. Since the stress testing was on the way it might had cascading effect on all the nodes.
Datagram Test result.
Tx summary 4 peers:
life: 96 MB/sec, 68774 packets/sec
now: 100 MB/sec, 71378 packets/sec, packets/burst: 1029, bursts/second: 69.41594
Success rate ranges from 0.87 to 1.0
Is a success rate less than 0.98 a cause for concern?
Regarding Size (to keep the extend client always connected to the proxy)
I do size request every 5 sec. if there is not other request has gone to the cluster in the past 5 sec.
In one of the oracle presentation it was said that "If you have map listener's and no other request from a long period of time extend client has to do a periodic size request to make sure that connection to proxy is alive"
In this regard my question is "Can the same effect achieved by doing member listener on the extend client and on disconnection do the size request?"
Proxy Size guide lines
In production number of extend clients are going to be ~2500 connected all the time.
Of which 2000 clients with few map listener + entry processor gets (Each of the clients will have distinct and disjoint set of data on which they work and listener)
And the remaining ~500 clients will do all kinds of request on the entire data set.
In this circumstance is there any recommendation for Number of extend-client : Number of extend-server : Number of threads ratio?
Regards
/Anand

OutOfMemoryError in cluster after TCP-extend client suspends processing.

Anybody able to explain why the following exception might occur... seems to occur when a client connected through tcp-extend is suspended during continuous query processing.
2007-12-20 21:19:49.009 Oracle Coherence GE 3.3.1/389 <Error> (thread=DistributedCache, member=1): Error sending MapEvent to Channel(Id=374515075, Connection=0x00000116F96C67F4A97BDC3A739C40D11DA1C36E3C50F9FBC8BB5AD8DCF4E16E, Open=true): java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:633)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:95)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at com.tangosol.coherence.component.comm.connectionManager.acceptor.TcpAcceptor$ByteBufferPool.instantiateResource(TcpAcceptor.CDB:7)

Hi,
We are also getting same excpetion. "java.lang.OutOfMemoryError: Direct buffer memory". I have changed the -XX:MaxDirectMemorySize=256m still getting the same Exception. Tried to investigate by getting the log level to 7 still could not figure out the exact issue. We are running on 3.3.1/389p4 version of Coherence. Only thing I noticed was at times there are many connections in netstat -a o/p in CLOSE_WAIT position.
Please Help!
Regards,
-Amit
This Exception we are getting on TCP Extend Proxy nodes.
2008-06-30 12:54:41.478 Oracle Coherence GE 3.3.1/389p4 <D6> (thread=DistributedCache, member=26): Outgoing ByteBufferPool increased to 266526720 bytes total
2008-06-30 12:54:41.478 Oracle Coherence GE 3.3.1/389p4 <D6> (thread=DistributedCache, member=26): Outgoing ByteBufferPool increased to 266536960 bytes total
2008-06-30 12:54:41.478 Oracle Coherence GE 3.3.1/389p4 <D6> (thread=DistributedCache, member=26): Outgoing ByteBufferPool increased to 266547200 bytes total
2008-06-30 12:54:41.478 Oracle Coherence GE 3.3.1/389p4 <D6> (thread=DistributedCache, member=26): Outgoing ByteBufferPool increased to 266557440 bytes total
2008-06-30 12:54:41.479 Oracle Coherence GE 3.3.1/389p4 <D6> (thread=DistributedCache, member=26): Outgoing ByteBufferPool increased to 266567680 bytes total
2008-06-30 12:54:41.479 Oracle Coherence GE 3.3.1/389p4 <D6> (thread=DistributedCache, member=26): Outgoing ByteBufferPool increased to 266577920 bytes total
2008-06-30 12:54:41.789 Oracle Coherence GE 3.3.1/389p4 <Error> (thread=DistributedCache, member=26): Error sending MapEvent to Channel(Id=1140491821, Connection=0x0000011AD950F44BAC1A65A1FB03F9B2AFAE8B5F9FF39688C81572023DF9F53B, Open=true): java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:632)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:95)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at com.tangosol.coherence.component.comm.connectionManager.acceptor.TcpAcceptor$ByteBufferPool.instantiateResource(TcpAcceptor.CDB:7)
at com.tangosol.coherence.component.comm.connectionManager.acceptor.TcpAcceptor$ByteBufferPool.acquire(TcpAcceptor.CDB:26)
at com.tangosol.coherence.component.comm.connectionManager.acceptor.TcpAcceptor$ByteBufferPool.allocate(TcpAcceptor.CDB:4)
at com.tangosol.io.MultiBufferWriteBuffer.advance(MultiBufferWriteBuffer.java:870)
at com.tangosol.io.MultiBufferWriteBuffer.<init>(MultiBufferWriteBuffer.java:32)
at com.tangosol.coherence.component.comm.connectionManager.acceptor.TcpAcceptor$TcpConnection.allocateWriteBuffer(TcpAcceptor.CDB:3)
at com.tangosol.coherence.component.comm.Connection.send(Connection.CDB:16)
at com.tangosol.coherence.component.comm.Channel.doSend(Channel.CDB:4)
at com.tangosol.coherence.component.comm.Channel.send(Channel.CDB:38)
at com.tangosol.coherence.component.net.extend.proxy.MapListenerProxy.onMapEvent(MapListenerProxy.CDB:9)
at com.tangosol.coherence.component.net.extend.proxy.MapListenerProxy.entryInserted(MapListenerProxy.CDB:1)
at com.tangosol.util.MapEvent.dispatch(MapEvent.java:191)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.DistributedCache$ViewMap$ProxyListener.dispatch(DistributedCache.CDB:22)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.DistributedCache$ViewMap$ProxyListener.entryInserted(DistributedCache.CDB:1)
at com.tangosol.util.MapListenerSupport$WrapperSynchronousListener.entryInserted(MapListenerSupport.java:856)
at com.tangosol.util.MapEvent.dispatch(MapEvent.java:191)
at com.tangosol.coherence.component.util.CacheEvent.dispatchSafe(CacheEvent.CDB:14)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.DistributedCache$BinaryMap.dispatch(DistributedCache.CDB:86)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.DistributedCache.onMapEvent(DistributedCache.CDB:31)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.DistributedCache$MapEvent.onReceived(DistributedCache.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onMessage(Service.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onNotify(Service.CDB:130)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.DistributedCache.onNotify(DistributedCache.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:35)
at java.lang.Thread.run(Thread.java:595)

Multiple TCP*Extend Proxy

Hi,
I am developing a C++ Client for Coherence and using the TCP*Extend to connect to the Cache.
I am able to run multiples nodes on same machine.
My goal is to add more machines to this setup (is different machine called cluster or node ?).
I would like to have a setup as explained below:
NamedCache: AQRCache
Cache Type: Near Cache (Local and distributed/remote)
Host_A:
Tcp*extend A
Node 1
Node 2
C++clientA running on Host_A talking to TCP*Extend_A.
Host_B:
Tcp*extend_B
Node 3
Node 4
C++clientB running on Host_B talking to TCP*Extend_B.
Host_C:
Tcp*extend_C
Node 5
Node 6
C++clientC running on Host_C talking to TCP*Extend_C.
Other C++ Clients:
C++clientD running on Host_D talking to TCP*Extend_A.
C++clientE running on Host_E talking to TCP*Extend_A.
C++clientF running on Host_F talking to TCP*Extend_A.
Questions:
1. To Add more machines, if I running one more instance on different machine with the same configuration file, will it work?
2. Is it possible to run seperate TCP*Extend Proxy on each Host and it will be part of the same cluster?
3. Or should all the C++ clients should talk to only one TCP*Extend Proxy?
Thanks,
NS

1. To Add more machines, if I running one more instance on different machine with the same configuration file, will it work?Yes, you can add more machines/nodes using the same configuration file.
2. Is it possible to run seperate TCP*Extend Proxy on each Host and it will be part of the same cluster?Yes, you can run multiple proxy nodes in a single cluster. Just about every production cluster has multiple proxy nodes.
3. Or should all the C++ clients should talk to only one TCP*Extend Proxy?No, there is no reason to force all C++ clients to connect to a single proxy. In fact you might consider configuring each client with the entire list of proxy servers. Each client will pick a server at random to connect to, thus ensuring that a single proxy isn't overloaded.
Also take a look at this document: http://coherence.oracle.com/display/COH35UG/Best+Practices+for+Coherence+Extend
Thanks,
Patrick

TCP Extend Errors

Hi All,
I am getting the below errors in my TCP-Extend Proxy nodes.
Oracle Coherence GE 3.5.3/465p3 <Error> (thread=Proxy:ExendTcpProxyService:TcpAcceptorWorker:7, member=19): Extend*TCP has determined that the connection to "Socket[addr=/11.11.56.123,port=18733,localport=17062]" must be closed to maintain system stability: This connection is 1 messages behind (112820921 bytes); the limit is 60000 messages (100000000 bytes).
What does this mean ? Any serious issues in the cluster ?

Hi user594809 ,
Coherence will close an Extend connection if it thinks that the client is not reading messages off of its queue fast enough as this can destabilise the extend proxy. The limit is 60000 messages behind or 100Mbytes behind.
In your case you have tried to return too much data in a single go (112,820,921 bytes) - I have seen this is often the result of an invocation service call or something similar. What tends to happen in this case is that your client has started to read the message back and while doing this Coherence closes the connection so you get stream exceptions in the client.
If you really need to return such a large amount of data there are a couple of settings you can add to your <tcp-acceptor> configuration on the client
For example:
<tcp-acceptor>
 <local-address>
 <address>localhost</address>
 <port>10000</port>
 </local-address>
 <limit-buffer-size>100000000</limit-buffer-size>
 <suspect-buffer-size>10000000</suspect-buffer-size>
</tcp-acceptor>The limit-buffer-size is the maximum size of the queue in bytes before the connection gets closed - The default is 100Mbytes
The suspect-buffer-size is the size of the queue that Coherence starts to log warnings - The default is 10Mbytes
If you increase the limit-buffer-size you should increase the suspect-buffer-size too.
Note there is a reason that these settings are there in the first place and changing them to allow very large results to be returned could have an adverse affect on your extend proxy node and your cluster so it is up to you to test things properly under load.
JK

TCP active open: Failed connect() Error: Connection timed out SMTP

Hi,
Messaging server version is,
./imsimta version
Sun Java(tm) System Messaging Server 6.2-6.01 (built Apr 3 2006)
libimta.so 6.2-6.01 (built 11:20:35, Apr 3 2006)
SunOS bglbbmr1-a-fixed 5.9 Generic_118558-28 sun4u sparc SUNW,Sun-Fire-V440
17-Dec-2008 10:47:40.08 1730.8e.741
tcp_local Q 4 [email protected] rfc822;[email protected] [email protected] /mta/queue/queue/tcp_local/013/ZUg0i1t9I0ZG~.00 <[email protected]>; TCP active open: Failed connect() Error: Connection timed out SMTP/xyz.my-domain.in
I have been getting this above error on my mail server from last
4-5 days. I am getting complaints from end users that the users can't
send any mails using Outlook but I did check with my test user I can
send mail by using webmail.
The Failed MX lookup Errors also getting in my logs the error detail given bellow.
17-Dec-2008 10:47:39.65 1730.91.737
tcp_local - Y TCP|0.0.0.0||209.85.143.114|25 SMTP/airtelmail.in/aspmx.l.google.com
17-Dec-2008 10:47:39.92 1754.41.255
tcp_notify - Y SMTP/infomedia18.in/infomedia18.in
17-Dec-2008 10:47:39.92 1754.41.256
tcp_notify Q 7 rfc822;[email protected] [email protected] /mta/queue/queue/tcp_notify/017/ZXg0i1t3U_ZoD.00 <[email protected]>; Failed MX lookup; try again later
17-Dec-2008 10:47:39.94 1754.41.257
tcp_notify Q 6 rfc822;[email protected] [email protected] /mta/queue/queue/tcp_notify/010/ZXg0i1t3U_ZoF.00 <0KBZ003MRGU7MQ30@my-domain> Failed MX lookup; try again later
I tried stopping and starting msg service using stop-msg and start-msg to sort out this above problem but no result. :(
When I do check the tcp_local queue it has been growing every day as well the tcp_notification queue also.
/opt/SUNWmsgsr/sbin/imsimta qm su
Messages
Channel Queued Size (Kb) Oldest
tcp_notify 10741 1080610.61 16 Dec, 00:59:24
tcp_local 8334 733849.31 15 Dec, 00:19:00
tcp_lmtpcn 0 0.00
tcp_be 0 0.00
reprocess 0 0.00
process 0 0.00
conversion 0 0.00
Totals 19075 1814459.92
This queues are increasing day by day.
One more thing is that I cant see a service/channel called CONVERSION running on my server when i do use this command.
ps -aef | grep conversion
root 6144 6000 0 11:14:28 pts/1 0:00 grep conversion
When i try to start it using imsimta qm utility, output shows as
qm.maint>; start conversion
QM-I-STARTED, channel was not stopped
qm.maint>;
Later I stopped and started conversion channel
qm.maint>; stop conversion
QM-I-STOPPED, channel stopped
qm.maint>; start conversion
QM-I-STARTED, channel started
qm.maint>;
I can see that on other servers the conversion channel is running and few msges are in queue. I do have other servers which running the same messaging server. But I am not getting why don't on this server. Where both servers having the same configuration.
Please, help me to sort out this issue.
Thanks in advance....
BSK

Thanks Mr. Shane,
The server which is running conversion channel.
ps -eaf|grep conversion
mailserv 16824 8472 3 17:08:11 ? 0:48 /opt/SUNWmsgsr/lib/conversion
mailserv 28728 8472 0 17:17:30 ? 0:00 /opt/SUNWmsgsr/lib/conversion
root 1057 26387 0 17:18:12 pts/1 0:00 grep conversion
more /opt/SUNWmsgsr/config/conversions
in-channel=*; in-type=application; in-subtype=*; in-disposition=*;
parameter-symbol-0=NAME; parameter-copy-0=*;
dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
message-header-file=2; original-header-file=1;
override-header-file=1; override-option-file=1;
command="/opt/SUNWmsgsr/private/virusscan.sh"
in-channel=*; in-type=x-zip-compressed; in-subtype=*; in-disposition=*;
parameter-symbol-0=NAME; parameter-copy-0=*;
dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
message-header-file=2; original-header-file=1;
override-header-file=1; override-option-file=1;
command="/opt/SUNWmsgsr/private/virusscan.sh"
in-channel=*; in-type=image; in-subtype=*; in-disposition=*;
parameter-symbol-0=NAME; parameter-copy-0=*;
dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
message-header-file=2; original-header-file=1;
override-header-file=1; override-option-file=1;
command="/opt/SUNWmsgsr/private/virusscan.sh"
in-channel=*; in-type=audio; in-subtype=*; in-disposition=*;
parameter-symbol-0=NAME; parameter-copy-0=*;
dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
message-header-file=2; original-header-file=1;
override-header-file=1; override-option-file=1;
command="/opt/SUNWmsgsr/private/virusscan.sh"
in-channel=*; in-type=video; in-subtype=*; in-disposition=*;
parameter-symbol-0=NAME; parameter-copy-0=*;
dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
message-header-file=2; original-header-file=1;
override-header-file=1; override-option-file=1;
command="/opt/SUNWmsgsr/private/virusscan.sh"
Following entry from /opt/SUNWmsgsr/lib/config-templates/imta_tailor
IMTA_CONVERSION_FILE=<msg.RootPathUNIX>/config/conversions
The server which doesnt show running conversion channel
#more /opt/SUNWmsgsr/config/conversions
!in-channel=*; in-type=*; in-subtype=*; in-disposition=*;
! parameter-symbol-0=NAME; parameter-copy-0=*;
! dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
! message-header-file=2; original-header-file=1;
! override-header-file=1; override-option-file=1;
! command="/opt/SUNWmsgsr/private/virusscan.sh"
in-channel=*; in-type=application; in-subtype=*; in-disposition=*;
parameter-symbol-0=NAME; parameter-copy-0=*;
dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
message-header-file=2; original-header-file=1;
override-header-file=1; override-option-file=1;
command="/opt/SUNWmsgsr/private/virusscan.sh"
in-channel=*; in-type=x-zip-compressed; in-subtype=*; in-disposition=*;
parameter-symbol-0=NAME; parameter-copy-0=*;
dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
message-header-file=2; original-header-file=1;
override-header-file=1; override-option-file=1;
command="/opt/SUNWmsgsr/private/virusscan.sh"
in-channel=*; in-type=image; in-subtype=*; in-disposition=*;
parameter-symbol-0=NAME; parameter-copy-0=*;
dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
message-header-file=2; original-header-file=1;
override-header-file=1; override-option-file=1;
command="/opt/SUNWmsgsr/private/virusscan.sh"
in-channel=*; in-type=audio; in-subtype=*; in-disposition=*;
parameter-symbol-0=NAME; parameter-copy-0=*;
dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
message-header-file=2; original-header-file=1;
override-header-file=1; override-option-file=1;
command="/opt/SUNWmsgsr/private/virusscan.sh"
in-channel=*; in-type=video; in-subtype=*; in-disposition=*;
parameter-symbol-0=NAME; parameter-copy-0=*;
dparameter-symbol-0=FILENAME; dparameter-copy-0=*;
message-header-file=2; original-header-file=1;
override-header-file=1; override-option-file=1;
command="/opt/SUNWmsgsr/private/virusscan.sh"
Following entry from /opt/SUNWmsgsr/lib/config-templates/imta_tailor
IMTA_CONVERSION_FILE=<msg.RootPathUNIX>/config/conversions
Is this above information u r asking?
As u wrote erlier, the conversion channel works some times and some times doesn't work.
Thanks lot...
BSKADAM

My Firefox no longer works, it will not go to any website and offers the message "The proxy server is rejecting connections". I had the same problem in IExplorer but was able to fix it. I did not choose a proxy server, ever. How do I fix this in Firefox?

I changed out my wireless router, from a Linksys to a Netgear. Everything seemed to work fine, but shortly afterward Firefox could not be used to find any websites, and gave me the message above. The same happened with Internet Explorer, but a tech representative for my ISP directed me to the page where "proxy server" was incorrectly checked and it fixed the problem. I don't know where to go in Firefox to find the same or a similar page. My laptop continued to work fine for some time after, but eventually developed the same problem. I uninstalled and reinstalled Firefox, with no luck, and all desktop shortcuts based on Firefox are inoperative. I just keep getting the message that the proxy server is rejecting connections. Since I did not select a proxy server, I don't know how to deselect it.

No idea what "tried everything" means.
Here is how you troubleshoot a password problem.
Go to your email providers web mail page and verify that you have a working username and password by logging into your account there. If you cannot log in there, look for the link to reset your password. Your email password is administered with your email provider.
Once you have a working password you need to remove any stored passwords in Thunderbird and replace with the new, working password.
From the menu bar select''' Tools-Options-Security-Passwords-Stored Passwords'''
Remove the passwords for the problem account. Close Thunderbird and restart it. You will be asked for a password. Enter the new password.
No menu bar with Tools? Press the '''alt '''key.
FYI: Thunderbird settings are stored in a different file from the program. Removing the program and reinstalling just keeps you busy for a little while and rarely fixes anything. As you have seen the old settings are picked up when Thunderbird is reinstalled.

SpamAssassin / Amavisd tuning - TCP active open: Failed connect() errors

I started this post as we were experiencing a problem with occassional delayed mail delivery to our mail store. As I continued writing it, I think that I've resolved it. But I wanted to post this anyway so if I'm doing things right it might help someone else; or, if I'm doing something wrong someone can correct me. :) For reference, we have a quad Sun v440, and process about 500,000 messages a day.
We were seeing messages like this in our log files:
09-Nov-2006 08:28:09.37 tcp_scan Q 1 [email protected] rfc822;[email protected] @tcp_scan-daemon:[email protected] TCP active open: Failed connect() Error: Connection timed out
example.com represents our local domain. The errors seemed to occur proportionally to the amount of incoming mail we received. That is, we saw more of them under load.
We are running SpamAssassin, Amavisd/clamav, and:
Sun Java(tm) System Messaging Server 6.2-5.01 (built Nov 22 2005)
libimta.so 6.2-5.01 (built 11:57:57, Nov 22 2005)
SunOS hostname 5.9 Generic_118558-11 sun4u sparc SUNW,Sun-Fire-V440
This line seems to say that the message is in the tcp_scan channel, and cannot send to 127.0.0.1:10024 (amavisd) because that port isn't listening.
My amavisd max_servers was set at 15. After reading http://www.ijs.si/software/amavisd/amavisd-new-magdeburg-20050519.pdf and making a wild guess, I increased it to 30. The errors were less frequent, but still occurred.
The output of 'sar -d' showed that my local disk was experiencing upwards of 80% utilization. I moved the amavisd temp directory and the spamassassin bayes db to a san volume, and that sped everything up dramatically. The connect errors went away, clamav avg time per message is at 4 seconds down from 10. I am now receiving errors from spamassassin that say:
Nov 9 10:27:27 hostname.example.com spamd[21601]: prefork: server reached --max-children setting, consider raising it
My --max-children is set to 25 right now, and the server is cpu bound under heavy load, so I see no reason to raise it.
So this is my setup. I am no longer having an immediate problem, but comments/questions are welcome.
Config files related to my setup are below.
Excerpts of imta.cnf:
! tcp_scan
[] $E$R${tcp_scan,$L}$U%[$L]@tcp_scan-daemon
! ims-ms
ims-ms defragment subdirs 20 notices 1 7 14 21 28 backoff "pt5m" "pt10m" "pt30m" "pt1h" "pt2h" "pt4h" maxjobs 2 pool IMS_POOL destinat
ionspamfilter1 fileinto $U+$S@$D
ims-ms-daemon
! tcp_local
tcp_local smtp mx single_sys remotehost inner switchchannel identnonenumeric subdirs 20 maxjobs 7 pool SMTP_POOL saslswitchchannel tcp
auth maytlsserver maysaslserver missingrecipientpolicy 0 aliasdetourhost tcpscan-daemon
tcp-daemon
! tcp_intranet
tcp_intranet smtp mx single_sys subdirs 20 dequeue_removeroute maxjobs 7 pool SMTP_POOL allowswitchchannel saslswitchchannel tcp_auth
maytlsserver maysaslserver missingrecipientpolicy 4 aliasdetourhost tcp_scan-daemon
tcp_intranet-daemon
! tcp_scan
tcp_scan smtp single_sys subdirs 5 notices 1 backoff "pt10m" "pt30m" "pt2h" "pt4h" dequeue_removeroute maxjobs 7 pool SMTP_POOL daemon
[127.0.0.1] port 10024
tcp_scan-daemon
option.dat:
SPAMFILTER1_LIBRARY=/opt/sunjes/SUNWmsgsr/lib/libspamass.so
SPAMFILTER1_CONFIG_FILE=/opt/sunjes/SUNWmsgsr/config/SpamAssassin
SPAMFILTER1_STRING_ACTION=data:,require ["addheader"]; addheader "Spam-test: $U"; require "fileinto"; fileinto "Junk";
SPAMFILTER1_OPTIONAL=1
dispatcher.cnf:
[SERVICE=SMTP-SCAN]
DEBUG=-1
PARAMETER=CHANNEL=tcp_scan
PORT=10025
IMAGE=IMTA_BIN:tcp_smtp_server
LOGFILE=IMTA_LOG:tcp_scan-server.log
STACKSIZE=2048000
INTERFACE_ADDRESS=127.0.0.1
SpamAssassin:
host=127.0.0.1
port=783
debug=0
mode=1
field=
verdict=Junk
USE_CHECK=0

Suggest dropping Amavis in preference to the built-in
integration, per our documentation. Simpler, more
performant.Sorry, my title wasn't very accurate. We are running SpamAssassin through the built in integration, and clamav through amavisd. There isn't any built in integration to clamav, is there?

TCP active open: Failed connect()

hi all,
i receive error from my email like
"TCP active open: Failed connect()x.x.x.x :25(No formatted text for errno = 111)"

Hi Mustapah,
if the error is in the email, then the hostname and/or IP address of the reporting server/appliance should be in there.
I suggest you try to look up the message in message tracking of your IronPort's GUI, this will tell you if the message was received by your appliance(s), and also if it was bounced on the appliance or on a different server.
Hope that helps,
Andreas

TCP active open: Failed connect() Error: Connection timed out

Hi,
There is one message which can't be delieved to the recepient with error " TCP active open: Failed connect() Error: Connection timed out" This is the error that we found in the mail.log_current file.
I have no problem sending email to the same address from my hotmail account.
any suggestion ?
Thanks,
Joe

The message can only mean one thing. Your server cannot connect to that host.
It's possible that the remote system is lying to your Messaging Server, and that's gotten it confused. Some firewalls do this by repeating the entire capability string from the server behind them, but can't actually handle all the capabilities.
PIX firewalls do this.
When you connect, they pretend to be able to "startls", but if you try, they hang up. They appear to handle "EHLO", but when you try, they hang up.
You can put debugging on the tcp_local channel, and see the actual conversation, if you like.
master_debug and slave_debug go into the imta.cnf definition line. MM_DEBUG=5 goes at the end of your option.dat file.
then
imsimta cnbuild
imsimta restart
will attempt to send the message. You will want to back the changes out, as you'll get lots of extra log files you really don't need.
Post the log file, and we can see what's happening.

Can this be done? Airport extreme connected wireless in extend mode to connect a wireless Mac Pro with an ethernet cable from the extended airport extreme..

Can this be done? Airport extreme connected wireless in extend mode to connect a wireless Mac Pro with an ethernet cable from the extended airport extreme.

This will work.....IF....you also have an Apple AirPort router configured to provide your wireless network.
In other words, you need to have two Apple AirPort routers to extend a wireless network.
When the AirPort Extreme is configured to "extend" the wireless of the "main" AirPort, it will provide more wireless coverage and the Ethernet ports are enabled, so you could connect your Mac Pro to any of the Ethernet ports and use that as a network and Internet connection.

TCP Extend (DefaultCacheServer rejects connections)

Similar Messages

Maybe you are looking for