Forward delay attribute in distributed queue

Dear experts,
I have a question in mind regarding this attribute/property which you can set inside the distributed queue in the admin console.
According to the help/documentation from WebLogic 10.3
This attribute (forward delay) has the following functionality:
The number of seconds after which a uniform distributed queue member with no consumers will wait before forwarding its messages to other uniform distributed queue members that do have consumers.
From the above interpretation, what i understand from the concept is that if a member in the distributed queue do not have any consumer, the messages in it will be forwarded to other members with consumers. However, what if one of the members which is already associated with a consumer, and the consumer is killed after a period of time, will the messages from that member be transferred to other members with consumers?
Since a member which with no consumer is not considered for message production.
Thanks.

However, what if one of the members which is already associated with a consumer, and the consumer is killed after a period of time, will the messages from that member be transferred to other members with consumers?Yes.
Tom

Similar Messages

  • Forward Delay on Uniform Distributed queues with no consumers

    Suppose you have a cluster with 2 members, and a uniform distributed queue targeted to the cluster. Messages are produced onto the queue by an application targeted to the cluster, so both distributed members should receive messages. Messages within the queues are consumed by a remote WL server which intermittantly connects ( with t3:// ).
    By default, the forward delay is disabled ( set to -1 ) on a queue, so no messages will be forwarded from a queue with no consumers to a queue that does have consumers. A load balancer in front of the cluster should result in the remote cluster's JMS consumer alternating its connection across the cluster.
    If the remote server is connecting to only a single node within the cluster, it will only consume messages from one queue, leaving the other queue's messages un-consumed. But if both queues are configured to use forward delay, and there are no consumers to either queue, will the messages be continuously forwarded back and forth between the queues?
    Or, once the forward delay has expired, will the queues wait for a consumer to show up and then forward all its messages at once?
    I'm wondering how much "sloshing" there's going to be between the nodes of the cluster, as messages are pushed back and forth between the queues.

    From the documentation, a distributed queue forward delay is:
    The amount of time (in seconds) that a distributed queue member with messages, but no consumers, will wait before forwarding its messages to other queue members that do have consumers.Therefore, messages will not forward if a consumer attaches before the configure idle period passes, or if there are no consumers on other queue members.
    I think one case where "Sloshing" would occur if all of the following were true: consumers are transitory and only exist for slightly more than the idle period, message rates are sufficiently high so that the consumer sometimes cannot keep up for significant periods of time, not all destinations are concurrently serviced by consumers, and the limited number of consumers that do exist frequently do not re-attach to the same member.
    Obligatory distributed queue best practice note: For the best performance, simplicity, HA, and message ordering support, it's generally a best practice to ensure that every member of a distributed queue is serviced by a consumer. WebLogic MDBs are specifically designed to handle this need simply and automatically.
    Tom
    Edited by: TomB on Apr 23, 2009 6:27 AM

  • OSB 10gR3 (WLS 10.3) - Distributed Queues & Load Balancing

    I have a question in relation to distributed queues and its JMS proxy service consumer in OSB
    I've set up a uniform distributed queue deployed using a sub-deployment resulting in the queue being targeted to the respective JMS servers in the cluster.
    I've then set up a messaging service using JMS as the transport with the following URI
    jms://server1:7011,server2:7012/weblogic.jms.XAConnectionFactory/myQueue
    When I look at the monitoring tab of my distributed queue, I can see 16 current consumers to one of the members but none for the other one. My understanding is that the proxy is just a mere MDB and as such I thought WLS was optimised to make sure all MDB instances would listen to all members of the distributed queue. Why do I have 16 consumers to one member only?
    Since only one member has consumers, any producer will always push messages to this member only. (I believe it is optimised to get a member with consumer(s) if any available)
    I've also tried to use a custom Connection Factory deployed the same way my distributed queue was, and ensure the connection factory had load balancing enabled. But no success with this either.
    jms://server1:7011,server2:7012/jms.MyConnectionFactory/myQueue
    I looked at the deployment - though not directly performed by me but rather the bus console - and it looks like the application is targeted to the cluster.
    How can I achieve true load balancing here, ensuring both members are consumed by my JMS proxy service?
    In that case, would any produced message go to either member then as both have consumers?
    Also, is the load balancing decision made by the producer when the Queue connection is created?
    If so, how do you achieve true load balancing? Do you need to ask for a new Q connection each time you want to send a message rather than caching the connection?
    Hope I am clear enough
    Thanks
    Arnaud

    This confused me too!
    The way I understand it, is that, as you say, a proxy service is like a single MDB. The MDB will bind to the queue it first finds when it connects.
    The URL that you specify which contains your two servers but the first address in the URL is the one that will be used for the connection. If the first server is unavailable, then the second one will be used.
    If you have a distributed queue, this doesn’t help much, as you do end up with one of the queue members with no consumers on it.
    You can configure a forward delay for the distributed queue, which will cause WLS to forward messages to a queue with consumers, but this isn’t a good idea if you have large JMS messages as WLS needs to serialize and de-serialize across the network to move the message.
    I think that what you have to do, is define two proxy services, one connecting to the first server, and the other connecting to the second.
    I haven’t found a better way so far, but it does seem a bit over the top, but then, if you wrote a an external java client which attached to a distributed queue you would specify the connection url and it would behave in the same way – if you wanted it to bind to both distributed destination members, you would have to code it or run two instances, so maybe its just working as it should – even though it seems strange.
    I think the producer will simply load balance across the distributed queue members, it doesn’t pay any attention whether there are to consumers attached – this happened to me the other day!!
    Pete

  • Uniform Distributed Queue - Delay in handing over to listener

    Hi,
    We have cluster of two managed servers. We have two physical destinations (queues) configured as uniform distributed queues. With this setup we observed that when a message is put in to the uniform distributed queue, it takes a while before it hands of to the consumer (message driven bean). This we are observing every time a message is delivered. Is there a configuration that we are missing that is causing this behavior?
    Thanks,

    Here are the answers
    How long is "a while"? How are you measuring the delay?
    It is usually around 5 mins. We are measuring the delay using wall clock timings (We have debug statements in the consumer. That are getting printed after 5 mins).Does your config.xml or JMS Module configuration have a "time-to-deliver-override/TimeToDeliverOverride" configured on a destination, or a "default-time-to-deliver/DefaultTimeToDeliver" configured on a connection factory, or does your app use the "setJMSDeliveryTime" extension?
    No. We have not overridden the any of these parameters. See below for jmsmodule.xmlDoes your sender or MDB app happen to have a "sleep( or wait(" encoded in it somewhere?
    No. :-)Did you enable transaction batching on the MDB?
    NoOr you hava you configured a batch delay somehow (this can happen if a SAF Agent or Bridge is in the flow)? No
    Are you using Error Destinations?
    No
    jms-module.xml
    <?xml version='1.0' encoding='UTF-8'?>
    <weblogic-jms xmlns="http://xmlns.oracle.com/weblogic/weblogic-jms" xmlns:sec="http://xmlns.oracle.com/weblogic/security" xmlns:wls="http://xmlns.oracle.com/weblogic/security/wls" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-jms http://xmlns.oracle.com/weblogic/weblogic-jms/1.1/weblogic-jms.xsd">
    <connection-factory name="inventoryWSQueueCF">
    <sub-deployment-name>inv_jms_server</sub-deployment-name>
    <jndi-name>inventoryWSQueueCF</jndi-name>
    <transaction-params>
    <xa-connection-factory-enabled>true</xa-connection-factory-enabled>
    </transaction-params>
    </connection-factory>
    <connection-factory name="inventoryTPQueueCF">
    <sub-deployment-name>inv_jms_server</sub-deployment-name>
    <jndi-name>inventoryTPQueueCF</jndi-name>
    <transaction-params>
    <xa-connection-factory-enabled>true</xa-connection-factory-enabled>
    </transaction-params>
    </connection-factory>
    <connection-factory name="inventoryWSQueueAlternateCF">
    <sub-deployment-name>inv_jms_server</sub-deployment-name>
    <jndi-name>inventoryWSQueueAlternateCF</jndi-name>
    <transaction-params>
    <xa-connection-factory-enabled>true</xa-connection-factory-enabled>
    </transaction-params>
    </connection-factory>
    <uniform-distributed-queue name="inventoryWSQueue">
    <sub-deployment-name>inv_jms_server</sub-deployment-name>
    <jndi-name>inventoryWSQueue</jndi-name>
    <load-balancing-policy>Round-Robin</load-balancing-policy>
    <forward-delay>10</forward-delay>
    </uniform-distributed-queue>
    <uniform-distributed-queue name="inventoryWSQueueAlternate">
    <sub-deployment-name>inv_jms_server</sub-deployment-name>
    <jndi-name>inventoryWSQueueAlternate</jndi-name>
    <load-balancing-policy>Round-Robin</load-balancing-policy>
    <forward-delay>10</forward-delay>
    </uniform-distributed-queue>
    <uniform-distributed-queue name="inventoryTPQueue">
    <sub-deployment-name>inv_jms_server</sub-deployment-name>
    <jndi-name>inventoryTPQueue</jndi-name>
    <load-balancing-policy>Round-Robin</load-balancing-policy>
    </uniform-distributed-queue>
    </weblogic-jms>
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  • Messages in Distributed Queue remain on one JMSServer in Cluster

    Hi,
              the situation is as follows:
              We're having Distributed Queues on a two-Node server setup. A queue therefore exists two times, once on each JMSServer on each node.
              Forward-Delay has been set on the DistibutedQueue to value 1 second (we tested with values 0 and 10, too).
              External Java client (both cluster nodes in the provider_url for initialcontext) connects to the DistributedQueue and sends messages. In the weblogic console you can see these messages on the physical queue on the JMSServer, let's call it QueueA on NodeA.
              An external java process to consume messages (again both clusters nodes in the jndi-connection-string) is started to consume messages. Because of round-robin it sometimes get's connections to QueueB (part of same DistributedQueue as QueueA, but on the other node in the cluster).
              When connected to QueueB on NodeB no single message is received. You can even see in the weblogic console that QueueA on NodeA has zero consumers and 50 messages, QueueB on NodeB has zero messages but one consumer.
              When we restart the consumer java process and it randomly get's connected to QueueA on NodeA it consumes fine, as it should.
              Should'nt Forward-Delay do exactly this? Transport messages from QueueA on NodeA with zero consumers to QueueB on NodeB which actually has consumers?
              Any help would be appreciated.
              Axel van Lil

    Hi,
              thanks for your answer. We opened a support case today for this issue... I don't know. Cluster seems to run fine, no warnings, no exceptions.
              The next step of this issue is the following (just to give an idea of our upcoming problems :-)
              External Java process listens on QueueA on NodeA (QueueA is part of a Distributed Queue). Currently receives all the messages located on that queue. We kill NodeA on that machine to simulate failover.
              The external process failovers immediately to NodeB (which is good! proven by netstat and debugger) but just doesn't receive messages even though there are messages in that queue! It seems that the new consumer is just being forgotten by the BEA server.
              I don't know... somehow our configuration is quirked... or the BEA implementation doesn't work at all...
              Rgds,
              Axel

  • Distributed queue usage with custom queue listeners in cluster

    Hi,
    Can someone help in solving the issue. We have a web application which will create consumers(threads in infinite loop) for an inbound queue(name comes from DB table) and after doing certain process result will be pushed to outbound queue(name comes from DB table).
    When application runs in single managed server(without clustering) every thing works fine.
    If I want to deploy the same application in two managed servers (with are under same cluster) which talks to same DB how to handle above scenario
    1. Create 2 JMS servers one on each managed server(on different machines)
    2. Create queue one for each JMS server
    3. Create distributed queue and add above two queues as members of it
    4. Create connection factory and target it to cluster
    After creating above instrastructure(similar infrastructure will be created for inbound and outbound queue), can I use distributed queue JNDI name to create listeners(threads) to read messages? Or do I need to create listeners for each individual member in the distributed queue?
    I know that while sending message to distributed queue, I can create sender on distributed queue and weblogic automatically takes care of sending the message to appropriate member in the DQ, but how it works for distributed queue?
    Also, want to know how can I access distributed queue? For example, if managed server 1 is accessed by http://10.11.134.24:7001 and managed server 2 is accessed by http://10.11.134.25:7001. which URL I need use in JNDI context provider URL to access distributed queue? can I use any URL will it make any difference?
    Generally for servlets/JSPs we create proxy server to access it through cluster, instead of directly using managed server URLs. But how it works for distributed queue?
    Thanks
    Santhosh
    Edited by: user8676720 on Oct 4, 2011 8:35 AM

              Yes.
              It turns out that if replicate local queue jndi name in cluster is set to false
              the message did not get forward :(
              Dongbo Xiao <[email protected]> wrote:
              >Sorry, the attribute is ForwardDelay, not ForwardPolicy.
              >
              >Dongbo
              >
              >Dongbo Xiao wrote:
              >
              >> Hi Eran,
              >>
              >> Have you set up the ForwardPolicy for the Distributed Queue?
              >> By default, the ForwardPolicy is -1, which turns off the forwarding.
              >>
              >> Dongbo
              >>
              >> eran wrote:
              >>
              >> > Hello,
              >> > I have 2 nodes in cluster each one has a pinned JMS server with a
              >queue QueueA
              >> > and QueueA2
              >> > respectively both mebers in a distributed destination dist_queue.
              >> > There is a consumer on QueueA only (startup class).
              >> > In case of a message sent to QueueA2 a consumer is created for QueueA2
              >automaticly
              >> > (seen in the admin console) out of the blue, and the message do not
              >get forward
              >> > to QueueA where my consumer is listening.
              >> > Monitoring JMS connections of both servers shows only my consumers'
              >connection.
              >> > Our application depends on the forward feature.
              >> > Am I doing something wrong? any ideas?
              >> >
              >> > Thanks.
              >> > Eran
              >
              

  • Cannot retrieve data in all members of my distributed queue

    I have a distributed queue with 3 members, queue member 1, queue member 2, and queue member 3.
    I triggered 2 events separately to go to the distributed queue, and it was successful as the 1st event triggered went to queue member 1 (physical queue1 – located on server 1) and the next event sent on the next trigger went to queue member 2 (physical queue 2 – located on server 2).
    The problem now is, after triggering a job that gets the event from distributed queue, only the event in queue member 1 was picked up. The job got completed without picking up the event in queue member 2 (different VM). The data in queue member 2 was left. The config file for the JMS Queue Connectivity may be seen below:
    jms.url=t3://<MS1_IP>:<MS1_PORT>,<MS2_IP>:<MS2_PORT>,<MS3_IP>:<MS3_PORT>
    jms.connfactory=ConnFactoryName
    jms.timeout=30000
    jms.queue=DistributedQueueName
    Note also that the connection factory is deployed to all managed servers as I could see the connection factory in the jndi tree of each managed server.
    Our job was able to get data from each of the managed server, one by one. An example I tried is I placed an event in queue member 1 then triggered the job, then the data was picked up. Then next, I placed the data in queue member 2 (with no other data in queue member 1 and 3), then the data in queue member 2 was picked up after the job run, and same when placed only in queue member 3.
    The main problem here is when there are data in all of the physical queues (queue member 1 to 3). As if the application is focused only to one physical queue, which shouldn’t be. Researching now how should it read all the members in the distributed queue.

    Hi,
    A comma separated URL list helps a client setup its initial JNDI connection into the cluster - it doesn't directly control load balancing of future JEE EJB, RMI, or JMS calls. By default, for remote clients, JEE calls will automatically load balance a new direct connection randomly among all servers in the cluster, even if the initial JNDI URL only references a single server's address. It's actually not truly random, as there may be internal heuristics that effect the load balance decision, plus the behavior is actually tunable, but, for almost all use cases, its best to stay with the default behavior and think of it as effectively random.
    WebLogic JMS load-balancing actually occurs at three layers: URL/DNS, "smart stub" connection creation (RMI, EJB, and JMS connection factory), and finally the distributed destination layers. For a full end-to-end explanation of load balancing at the different layers of the WL stack, I highly recommend the JMS chapter of the book "Professional Oracle WebLogic".
    What's happening in your case is that JMS individual consumer always load balances to one particular queue member, and then sticks to that queue member for their lifetime. This is expected behavior for consumer load balancing, and is fully documented in the distributed destination documention of the JMS programmer's guide.
    If you want to ensure full coverage of a distributed queue, in order of highest to lowest preference:
    * Most highly recommend: Use a WL MDB or a 11gR1PS3 SOA Adapter (if you happen to be using Oracle's SOA layered product). These automatically ensure that all distributed queue members are serviced by at least one dedicated consumer.
    * Ensure you have more sessions+consumers than queue members. Periodically refresh connections/sessions/consumers in order to ensure that newly started members are serviced even when all consumers are already connected (eg, code your clients to reconnect every few minutes).
    * If your application is not performance sensitive, configure the "queue forward delay" setting on a distributed queue. Messages that are trapped on a queue member with no consumers for longer than this delay will be automatically forwarded a member that has consumers.
    * Least recommended, very advanced usage only: Code clients to use the new 10.3.4 public "destination availability" extensions. These extension APIs provide a notification service for distributed destination member up/down events.
    Regards,
    Tom

  • Forward Delay not working

    I have a distributed queue, whose subdeployments points to two JMS Servers each having there own persistent DB store. Now i have configured forward delay of 5 (implying 5 seconds) on the Distributed Queue. So essentially if there are no consumers for a queue for 5 seconds the messages should be forwarded to other member of the distributed queue but the same is not happeneing, the messages never get forwarded to other member in the distributed queue? What could be the problem?

    I have already checked the JNDI tree and those show the distributed members correctly.
    As you said as of weblogic 9.2 its not queue forward is not encoraged then how come MDBs are able to listen from both queue members.
    Eg: Say you have two queue emmebr but when you start your servers only one is accessible so all consumers will listen to queue member QA availbel on jms server JA having database persistent store DA, the other queue member QB available on JMS server JB having database persistent store DB will not be listened to even after the seconds managed server on whch JB is deployed comes back?Yes - MDBs handle all cases automatically.
    When an MDB runs on a different cluster/server than its source distributed queue, it automatically creates at least one active consumer on each distributed queue member.
    When an MDB is targeted to the same cluster as its source distributed queue, it automatically creates a deployment for each queue member, or, in very limited situations, you may want to configure a behavior that causes each local deployment to connect to every queue member.
    The MDB code automatically dynamically handles all cases of changes in member availability such as shutdown, start, new configured members, deleted members, and the migration of members from server to server.
    I am not sure if you have can two JMS servers using the same DB Store?No. No two JMS servers can share a DB store unless the JMS servers and DB store all share the same target (and therefore all run on the same WebLogic Server). Even then, the two JMS servers will not manipulate or expose each-other's messages -- they will only expose their own messages.

  • JMSAn error occurred while forwarding a message for distributed destination

    Hi,
    i have getting issue with JMS Server delivery fail, below is the errors
    ####<Aug 24, 2012 2:03:54 PM CDT> <Warning> <JMS> < <[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default
    (self-tuning)'> <<WLS Kernel>> <BEA1-0979336B7FF44582D27D> <> <1345835034568> <BEA-040498> <An error occurred while forwarding a message for distribute
    d destination member SystemModule-0!d4a JMS Server@D Distributed Topic: weblogic.messaging.dispatcher.DispatcherException: java.rmi.RemoteExcepti
    on: Could not establish a connection with -2037322150112737172S:10.110.7.114:[8011,8011,-1,-1,-1,-1,-1]:d:d4a, java.rmi.ConnectException: Destina
    tion unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Could not establish a connection with -2037322150112737172S:10.110.7.114:[8011,8011,-1,-1,-1,-1,-1]:d:d4a, jav
    a.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination
    weblogic.messaging.dispatcher.DispatcherException: java.rmi.RemoteException: Could not establish a connection with -2037322150112737172S:10.110.7.114:[
    8011,8011,-1,-1,-1,-1,-1]:d:d4a, java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Could not establish a connection with -2037322150112737172S:10.110.7.114:[8011,8011,-1,-1,-1,-1,-1]:d:d4a, jav
    a.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination
    at weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchAsync(DispatcherWrapperState.java:155)
    at weblogic.jms.dispatcher.DispatcherAdapter.dispatchAsync(DispatcherAdapter.java:84)
    at weblogic.jms.backend.BEForwardingConsumer$1.run(BEForwardingConsumer.java:492)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
    at weblogic.jms.backend.BEForwardingConsumer.processMessages(BEForwardingConsumer.java:488)
    at weblogic.jms.backend.BEForwardingConsumer.pushMessages(BEForwardingConsumer.java:300)
    at weblogic.messaging.util.DeliveryList.run(DeliveryList.java:256)
    at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
    java.rmi.RemoteException: Could not establish a connection with -2037322150112737172S:10.110.7.114:[8011,8011,-1,-1,-1,-1,-1]:d:d4a, java.rmi.Con
    nectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Could not establish a connection with -2037322150112737172S:10.110.7.114:[8011,8011,-1,-1,-1,-1,-1]:ds:ds4a, jav
    a.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    ####<Aug 24, 2012 2:03:54 PM CDT> <Warning> <JMS> <dal604se113com> <a> <[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default
    (self-tuning)'> <<WLS Kernel>> <BEA1-0979336B7FF44582D27D> <> <1345835034568> <BEA-040498> <An error occurred while forwarding a message for distribute
    d destination member SystemModule-0!d4a JMS Server@D Distributed Topic: weblogic.messaging.dispatcher.DispatcherException: java.rmi.RemoteExcepti
    on: Could not establish a connection with -2037322150112737172S:10.110.7.114:[8011,8011,-1,-1,-1,-1,-1]:ds:ds4a, java.rmi.ConnectException: Destina
    tion unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Could not establish a connection with -2037322150112737172S:10.110.7.114:[8011,8011,-1,-1,-1,-1,-1]:dsa, jav
    a.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination
    weblogic.messaging.dispatcher.DispatcherException: java.rmi.RemoteException: Could not establish a connection with -2037322150112737172S:10.110.7.114:[
    8011,8011,-1,-1,-1,-1,-1]:ds:ds4a, java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Could not establish a connection with -2037322150112737172S:10.110.7.114:[8011,8011,-1,-1,-1,-1,-1]:ds:ds4a, jav
    a.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination
    at weblogic.messaging.dispatcher.DispatcherWrapperState.dispatchAsync(DispatcherWrapperState.java:155)
    at weblogic.jms.dispatcher.DispatcherAdapter.dispatchAsync(DispatcherAdapter.java:84)
    at weblogic.jms.backend.BEForwardingConsumer$1.run(BEForwardingConsumer.java:492)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
    at weblogic.jms.backend.BEForwardingConsumer.processMessages(BEForwardingConsumer.java:488)
    at weblogic.jms.backend.BEForwardingConsumer.pushMessages(BEForwardingConsumer.java:300)
    at weblogic.messaging.util.DeliveryList.run(DeliveryList.java:256)
    at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
    java.rmi.RemoteException: Could not establish a connection with -2037322150112737172S:10.110.7.114:[8011,8011,-1,-1,-1,-1,-1]:ds:ds4a, java.rmi.Con
    nectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Could not establish a connection with -2037322150112737172S:10.110.7.114:[8011,8011,-1,-1,-1,-1,-1]:ds:ds4a, jav
    a.rmi.ConnectException: Destination unreachable; nested exception is:
    java.net.SocketException: Connection reset; No available router to destination; nested exception is:
    java.rmi.ConnectException: Destination unreachable; nested exception is:
    Edited by: Ikhan on Aug 29, 2012 4:00 AM
    Edited by: Ikhan on Aug 29, 2012 4:01 AM

    The distributed topic forwarders replicate messages to every member of a "replicated distributed topic". They will report errors if intra-cluster communication is interrupted or if servers in the cluster shut down. I recommend checking your server logs to see if other errors/warnings exist, and, if you suspect an intra-cluster communication problem, consulting the cluster troubleshooting guide(s) in the edoc.
    Most newer apps (apps hosted on versions 10.3.4 or higher) should consider using the new "partitioned distributed topics" instead of "replicated distributed topics". These don't use forwarders and scale better.
    HTH,
    Tom

  • Looking up a distributed queue with two persistent stores using two JMS Svr

    I am trying to do the following:
    1. Setup a distributed queue, i have two servers which are part of the cluster:
    Server A: t3://localhost:7003
    Server B: t3://localhost:7005
    I go in and create two jms servers:
    JMS Server JA: Target on Server A
    JMS Server JB: Target on Server B
    Now as the jms servers need to use a seperate persistent store for each one of them i create two persistent stores.
    2. Now from my MDB which is deployed on another server i lookup the queue using
    t3://localhost:7003,localhost:7005 as the provider url
    My problem is that i always end up listening to the messages on the first jms server and never get to read the messages on jms server JB as i guess i am able to connect to JMS Server JA so i never try and connect to JB? What to do about this?
    Edited by: user4828945 on Mar 23, 2009 2:32 PM

    Allocation of consumers wouldn't take into account the number of messages on the queue - they'd be allocated randomly. The scenario you're proposing shouldn't happen though - WebLogic Server takes into account whether a member has consumers when sending to a distributed destination, but otherwise, assuming that Queue 1 and Queue 2 both have consumers, then distribution of load will be equal. It's not the amount of consumers that determine how many messages get sent to a distributed destination member - it's whether it has members at all.
    Assuming that did occur initially though, you'd expect processing to be a little bit more intensive on the server with the queue holding 30 messages. It would pretty quickly even up though.
    From that point forward, it would be somewhere between difficult and impossible to get to the second scenario, where you have an unequal number of messages in each distributed destination member, unless the work being sent with each message to an MDB can vary significantly in how long it would take to process.
    Assuming (and it's a big if) you could get to that scenario, then the MDBs wouldn't switch over - they stay connected to a particular distributed destination member. And it's their connection to a member as a consumer that controls how WebLogic Server load balances messages (assuming default configuration) so that's part of what makes it unlikely to get there.
    From going back to first principles in the documentation, it seems like your best result would actually be from deploying the MDB to the cluster - that way, there's no remote connections to JMS queues, and you get a pool of MDBs on each server instance.
    Ref here: http://e-docs.bea.com/wls/docs81/ejb/message_beans.html

  • OSB distributed queue proxy configuration

    I have set up a distributed queue in WeblogicServer and I wonder how should I configure OSB proxy to consume JMS messages from it. I have read documentation at Oracle site and it says a distributed queue groups a bunch of local JMS queues in different servers that can be accesed through a common JNDI name. This provide load balancing and failover to messages put in the queue.
    In OSB proxy a JMS destination is configured with following URL: jms://server:port/destination. So if I have a distributed queue that maps to two JMS servers, what server and port must I set in proxy URL? Do I need to setup two different proxies, one for JMS server1 and another for JMS server2?
    Regards,
    DC.

    For weblogic JMS message producer load balancing happens at 3 different places.
    1- For intial context lookup - this depends upon the URI you specify in the Initial Context Call. e.g. if you specify t3://localhost:7002,localhost:7003 , all context lookups happen on port 7002. Port 7003 is used only if 7002 is down. Thus this supports failover instead of load balancing. If you want load balancing for context lookup then you have to use a dns based cluster address where dns does the name resolution to a different address each time.
    2- When a jms connection is created. The jms connection can be created to any managed server to which the Connection factory is targeted to. Thus you can have your context lookup on 7002 , but the actual jms connection can get created on 7003. If server affinity is enabled for the connection factory then the jms connection will be created to the same managed server instance as on which context lookup happened.
    3- When a message Producer send is executed. The message send can land on any dd member in the cluster if load balancing is enabled. If server affinity is enabled, then the message will end up on dd member on the same managed server instance to which the jms connection was made. For e.g. assume you have your jms client app has a jms connection on ms1 and the message produced ends up on ms2, then the following path would have been taken.
    jms client app --------Over jms connection-------> ms1 -------internally forwards ----------> ms2 ------puts message ---> dd2
    I would recommend you to read the JMS chapter of Professional Oracle Weblogic Server book where this is explained clearly.

  • Redirecting failed messages to other consumers of distributed queue

    Hi,
    We have a simple cluster with two servers A and B, each hosting one MDB whose task is to consume message from a distributed queue and forward them to an EIS via a JCA resource adapter. Server B acts as a failover server. If the resource adapter is unable to deliver the message, the MDB will throw a runtime exception and the message will therefore be redelivered. In order to achieve high-availability, we have configured the distributed queue to send the redelivered message to an error destination (dead-letter-queue DLQ). The DLQ again has another instance of the MDB as a consumer with a different resource adapter.
    Using DLQs is working fine as a high-availability mechanism when the MDB is unable to deliver the message to the EIS, but is this a common or valid approach?
    An alternative to using DLQs would be to configure weblogic to send redelivered message to other consumers on the distributed queue. The problem we have is that the failed message always gets redelivered to the same MDB (i.e. the MDB which consumes messages but throws an exception because the resource adapter fails). Is it somehow possible to configure the MDB or even change the MDB code to notify weblogic to send the failed message to another consumer of the distributed queue? Would the MDB be able to disconnect the JMS connection when throwing the exception and if so, would the disconnection cause the application server to deliver the message to another consumer?
    Many thanks

    Thanks Tom,
    Setting the distributed-destination-connection property to EveryMember seems to be exactly what we need to allow other distributed queue member to consume the message which has been put back on the queue after a rollback. In order to ensure that it won't be the same MDB consuming the failed message again, we would have to temporarily suspend the MDB. From what I read, one approach is to sleep the MDB after throwing the runtime exception (but is this possible, i.e. is it not going to interrupt the onMessage before going to sleep?). I also read that there is a new property from WLS 9.0 onwards to automatically stop the MDB for a certain time after an exception has ocurred, how can I configure this? Are we also going to have to set the MDB pool size to 1 in order to ensure that the message gets consumed by an MDB on a different server?
    We have also tried to use a non-collocated approach instead using a collocated approach with distributed queues but we end up with the same problem that a message does not get redelivered to MDBs on other servers after a runtime exception has been thrown.
    Thanks a lot

  • Using distributed queues and factories in a weblogic cluster

    Hi all,
    in our environment we have a domain which has two machines. One is running the AdminServer and Node1 and the 2nd machine is running Node2.
    For each Node I have created a JMS Server and a corresponding file store located at each machine.
    I've also created a JMS Module containing our required queues, factories and topics. Queues and Topics are of the Uniform Distributed type.
    I've targeted this module at the cluster and experimented with creating a subdeployment. But from what I recon, since all my resources have default targeting
    enabled, a subdeployment is not required, they target what is targeted by the JMS Module, right?
    However I get an exception when both nodes are up:
    <Nov 4, 2008 11:47:01 AM EET> <Warning> <JMS> <BEA-040498> <An error occurred while forwarding a message for distributed destination member       
    MyJMSModule!MyJMSServer@myTopic: weblogic.messaging.dispatcher.DispatcherException: java.rmi.RemoteException: Couldn't connect to weblogic.rjvm.RJVMImpl@187766d - id: '-1415064081927280896S:10.0.0.5:
    [7001,7001,-1,-1,-1,-1,-1]:app1:7003,app2:7001:my_domain        :NodeApp2' connect time: 'Tue Nov 04 11:47:01 EET 2008' - it is likely that the connection has already been shut down; nested exception is:
    5564732         java.rmi.ConnectException: Couldn't connect to weblogic.rjvm.RJVMImpl@187766d - id: '-1415064081927280896S:10.48.92.70:[7001,7001,-1,-1,-1,-1,-1]:app1:7003,app2:7001:my_domain:NodeApp2'
    connect time: 'Tue Nov 04 11:47:01 EET 2008' - it is likely that the connection has already been shut downThe two nodes are up and running and the node managers as well. Is this the correct way to create the queues and topics for a clustered environment?
    Thanks
    Edited by: dvm on Nov 4, 2008 3:12 AM

    I've done the steps but the problem still persists. I don't know if I had this before, but now both machines can't see each other.
    From Node1 logs *.out file:
    <Warning> <JMS> <BEA-040498> <An error occurred while forwarding a message for distributed destination member MyJMSModule!MyJMSServer2@my_cacheTopic:
    weblogic.messaging.dispatcher.DispatcherException: java.rmi.RemoteException:
    Couldn't connect to weblogic.rjvm.RJVMImpl@62017d - id: '4786464166486913267S:10.48.92.70:[7001,7001,-1,-1,-1,-1,-1]:app1:7003,app2:7001:my_domain:NodeApp2'
    connect time: 'Wed Nov 05 16:03:53 EET 2008' - it is likely that the connection has already been shut down; nested exception is:
    java.rmi.ConnectException: Couldn't connect to weblogic.rjvm.RJVMImpl@62017d - id: '4786464166486913267S:10.48.92.70:[7001,7001,-1,-1,-1,-1,-1]:app1:7003,app2:7001:my_domain:NodeApp2'
    connect time: 'Wed Nov 05 16:03:53 EET 2008' - it is likely that the connection has already been shut downand from Node2 logs *.out file:
    <Warning> <JMS> <BEA-040498> <An error occurred while forwarding a message for distributed destination member MyJMSModule!MyJMSServer1@my_cacheTopic:
    weblogic.messaging.dispatcher.DispatcherException: java.rmi.RemoteException:
    Couldn't connect to weblogic.rjvm.RJVMImpl@24b64a - id: '-788798942991630857S:10.48.92.69:[7003,7003,-1,-1,-1,-1,-1]:app1:7003,app2:7001:my_domain:NodeApp1'
    connect time: 'Wed Nov 05 16:03:59 EET 2008' - it is likely that the connection has already been shut down; nested exception is:
    java.rmi.ConnectException: Couldn't connect to weblogic.rjvm.RJVMImpl@24b64a - id: '-788798942991630857S:10.48.92.69:[7003,7003,-1,-1,-1,-1,-1]:app1:7003,app2:7001:my_domain:NodeApp1'
    connect time: 'Wed Nov 05 16:03:59 EET 2008' - it is likely that the connection has already been shut downThis happens just after the managed servers have been started with the application previously deployed.

  • Programming for Using JMS Distributed Queues

    Hi,
    Does anyone know specifically how your meant to write java code in order to fully make use of JMS Distributed queues. At the moment we have a 3rd party app, which I don't think is written correctly, as it always locks up after using a distributed queue, or a consumer is at 0 on one of the members of the distributed queue, when it should always should be constant and uniform.
    Its as if their code hasn't been written correctly.

    Distributed queue applications are encouraged to leverage WebLogic MDBs, which automatically ensure that consumers are attached to each distributed queue member, and to write applications in a way that has no dependency on which JVM a particular message is processed.
    At a guess, the third party product is setting up a consumer on one member of the distributed Q, while messages are being produced to the other member. If this assumption is true, here are some thoughts:
    * Consider using a single Q for the third party product's usage instead of a distributed queue. Configuring a distributed queue is not adding any high availability, as the product is expecting all messages to pin to the same Q instance.
    * Producers and consumers load-balance independently, but, when "affinity" and "load balance" are both configured to false on a Producer's connection factory, producers will have a tendency to load balance to the member that hosts more than one consumer. This might help somewhat, but my guess is that it won't help in all cases -- for example, after a restart I'm not sure that there's a guarantee that the consumer won't load balance to the "wrong" member (the member that has no messages).
    * It might help to enable "queue forwarding" on the distributed queue configuration. This feature automatically forwards messages trapped in a destination that has no consumers to a queue that has consumers. "Queue forwarding" is not compatible with WebLogic's "unit-of-order" feature, but it's likely the third party product isn't using this feature. Queue forwarding also imposes a performance penalty - but whether you notice the difference depends on whether you have high performance requirements.
    Hope this helps,
    Tom
    Edited by: TomB on Mar 9, 2011 6:28 PM

  • How to process one message at a time on a distributed queue

    I have a situation where I need to execute/process a task of a type
              asynchronously and this should be done one at a time. To do that I am thinking of setting
              up a distributed queue (for load balancing and failover). Now I have
              this requirement that only one task should be processed at one time. So
              for example if two messages (m1 followed by m2) are sent to distributed
              queue and I have the distributed queue which points to two physical
              queues on two different instances of weblogic(wl1 and wl2), the
              distributed queue will likely send m1 to wl1 and m2 to wl2 (assuming
              distributed queue is setup with round robin policy). If m1 has started
              executing/processing in my MDB on wl1, I do not want start processing
              m2 on wl2 untill m1 has been processed by wl1. Is it possible to do
              this on weblogic 8.1 JMS?
              If this is not supported I was thinking of implementing a database
              based locking framework and each mdb will use that framework to check
              if a task of a particular type is being processed before processing a
              new task. Also in this case I will need to use max. 1 mdb for each
              queue on each weblogic server.
              --Navjeet                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

    Hello navjeet,
              I don't think there are any parameters that you can set on a distributed queue to ensure messages are handled in the sequential manner you describe so it looks like you will have to implement your own locking mechanism.
              In reposponse to your other question you can set the number of MDbeans in the free pool to 1 (or the number of queues you have) in the MDBean deployment descriptor.
              see:
              http://e-docs.bea.com/wls/docs81/ejb/message_beans.html#1120592
              Cheers,
              Hoos

Maybe you are looking for