JMS messages stuck as Pending messages
Hi.
I am trying to write an application to process huge amounts of data. I try to
process around 1,000, 000 messages [java objects] using a single queue and 8 MDB.
The strange thing is that in the end there are always 10 messages stuck in the
pending message category.
I can also observe that a weblogic thread is stuck. [can see it in the console]
The stuck thread current description is something like "Connectio27.Session28".
I have tried to use the QueueConnectionFactory from a EJB using resource reference
instead of fetching it directly. Even this has not helped. I guess the problem
is that while loading the messages onto the queues, one of the threads gets stuck
somewhere.
I am sing MDB's for processing. There doesnot seem to be any problems on the MDB
side.
If I reboot the server the 10 messages get processed successfully.
Any ideas will be highly appreciated.
-Sharma
if you waited for a minute or so between each threaddump, it definitly
sounds like an issue around the database query not returning or there is too
much data being returned.
sree
"Ghanshyam Sharma" <[email protected]> wrote in message
news:40e2d9d6$1@mktnews1...
>
> Hmm, I didnt wait too much between thread dumps. Probably a minute or two.
>
> Let me check the oracle stuff.
>
> Thanks a Lot.
>
> -Sharma
>
> "sree bodapati" <iwillreply@thenewsgroups> wrote:
> >Threaddumps dont seem to show much except for it looks like both execute
> >thread 13 and 12 are stuck reading from oracle , you may want to see
> >what
> >query is being sent to oracle by instrumenting you MDB code see that
> >query
> >properly returns. The threads are runnable but they dont seem to move
> >, what
> >was the duration between each thread dump ?
> >
> >/
> >sree
> >
> >
> >"Ghanshyam Sharma" <[email protected]> wrote in message
> >news:40e2ce39$1@mktnews1...
> >>
> >> Hi Tom.
> >>
> >> I am attaching a thread dump of the server on windows, when I have
> >20
> >messages
> >> stuck as pending with MDB's having not-supported transaction attribute.
> >>
> >> I hope this helps in debugging the problem. Please let me know if
> >something is
> >> needed.
> >>
> >> Thanks everyone.
> >> -Sharma
> >>
> >>
> >> Tom Barnes
>
><[email protected].bea.com>
> >> wrote:
> >> >As I wrote below, a thread-dump of the WL Server that hosts
> >> >the stuck MDB might help explain things.
> >> >
> >> >Sharma wrote:
> >> >
> >> >> Hm, this is confirmed. I pumped in one million messages a few time
> >> >with and without
> >> >> file stores to differe the performance, but every time I get a clean
> >> >slate :)
> >> >>
> >> >> Looks like having required transaction attribute somehow removes
> >this
> >> >problem.
> >> >>
> >> >>
> >> >> But still I do not understand why messages were getting stuck with
> >> >not-supported
> >> >> attribute. Tom is it possible to understand this. The
> >phenomenon/Reason
> >> >behind
> >> >> this may show up somewhere else :)
> >> >>
> >> >> -Sharma
> >> >>
> >> >> Tom Barnes
>
><[email protected].bea.com>
> >> >> wrote:
> >> >>
> >> >>>The default is no transactions. I also find it curious
> >> >>>that this helps. I wonder if tx timeouts are helping here...
> >> >>>
> >> >>>Sharma wrote:
> >> >>>
> >> >>>
> >> >>>>Surprisingly I could process all one million messages if I
configure
> >> >>>
> >> >>>the MDB's
> >> >>>
> >> >>>>as folowing.
> >> >>>>
> >> >>>>Default-transaction=required
> >> >>>>Transaction-typr=container
> >> >>>>
> >> >>>>When I was getting the problem, I had nothing for these settings.
> >> >I
> >> >>>
> >> >>>do not know
> >> >>>
> >> >>>>what is the default, probably Not-supported ???
> >> >>>>
> >> >>>>Let me try my testing a few times and than confirm to you guys.
> >> >>>>
> >> >>>>Tom Barnes
>
><[email protected].bea.com>
> >> >>>>wrote:
> >> >>>>
> >> >>>>
> >> >>>>>Since you narrowed the problem down to the MDB side, can
> >> >>>>>post a thread dump of the server when it is stuck? That
> >> >>>>>might help diagnose the problem.
> >> >>>>>
> >> >>>>>Ghanshyam Sharma wrote:
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>>Infact The thread which is stuck is not the loader/message
producer
> >> >>>>>
> >> >>>>>thread. It
> >> >>>>>
> >> >>>>>
> >> >>>>>>is a thread handling message driven beans. My producers end up
> >populating
> >> >>>>>
> >> >>>>>the
> >> >>>>>
> >> >>>>>
> >> >>>>>>JMS correctly.
> >> >>>>>>
> >> >>>>>>This is 81sp02.
> >> >>>>>>-Sharma
> >> >>>>>>
> >> >>>>>>"Ghanshyam Sharma" <[email protected]> wrote:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>>Hi.
> >> >>>>>>>
> >> >>>>>>>I am trying to write an application to process huge amounts
> >of
> >> >data.
> >> >>>>>>>I try to
> >> >>>>>>>process around 1,000, 000 messages [java objects] using a single
> >> >>>
> >> >>>queue
> >> >>>
> >> >>>>>>>and 8 MDB.
> >> >>>>>>>The strange thing is that in the end there are always 10
messages
> >> >>>
> >> >>>stuck
> >> >>>
> >> >>>>>>>in the
> >> >>>>>>>pending message category.
> >> >>>>>>>
> >> >>>>>>>I can also observe that a weblogic thread is stuck. [can see
> >it
> >> >in
> >> >>>>>
> >> >>>>>the
> >> >>>>>
> >> >>>>>
> >> >>>>>>>console]
> >> >>>>>>>The stuck thread current description is something like
> >"Connectio27.Session28".
> >> >>>>>>>
> >> >>>>>>>I have tried to use the QueueConnectionFactory from a EJB using
> >> >resource
> >> >>>>>>>reference
> >> >>>>>>>instead of fetching it directly. Even this has not helped. I
> >guess
> >> >>>>>
> >> >>>>>the
> >> >>>>>
> >> >>>>>
> >> >>>>>>>problem
> >> >>>>>>>is that while loading the messages onto the queues, one of the
> >> >threads
> >> >>>>>>>gets stuck
> >> >>>>>>>somewhere.
> >> >>>>>>>
> >> >>>>>>>I am sing MDB's for processing. There doesnot seem to be any
> >problems
> >> >>>>>>>on the MDB
> >> >>>>>>>side.
> >> >>>>>>>
> >> >>>>>>>If I reboot the server the 10 messages get processed
successfully.
> >> >>>>>>>
> >> >>>>>>>Any ideas will be highly appreciated.
> >> >>>>>>>-Sharma
> >> >>>>>>
> >> >>>>>>
> >> >>
> >> >
> >>
> >
> >
>
Similar Messages
-
Hi,
We have Weblogic JMS provider and we use spring JMS listener to read messages of the queues. Our Spring listeners are running inside a webMethods Integration Server container. For a few of the queues, we have been seeing messages that are stuck in the "Pending" state on the JMS queues. The state string inside webLogic is shown as "receive". We don't have any evidence from our logs that the message actually is received by the listener or by subsequent processes down stream. The messages gets cleared when the Weblogic server is restarted.
Has anyone faced a similar issue or any ideas as to what might be causing this stuck messages?
Our weblogic version is 9.2
Regards,
Aneel
Edited by: user3127831 on Mar 24, 2009 7:02 AMHi Aneel
Typically JMS messages get stuck in pending state between components because of XA issues - due to configuration on one side of the fence or another, transactions aren't committed successfully, so a JMS message is placed on a queue successfully, but is not available due to the transaction that created it not committing successfully.
What is creating the messages being placed on the queues? What's the difference between how you're creating messages for the queues you can read successfully and the ones that are not becoming available? -
JMS Messages getting stuck in queues
Hi,
I am facing this peculiar issue with queue messages not getting picked up.
Application Architecture: There is 1 JMS queue ( which resides in WLDomain1 with in a Unix Box) and there are 2 windows boxes having 2 weblogic domains each with 3 instances of MDB deployed on each of the domains. So total of 12 consumers (2 Box * 2 Domain * 3 Instances of MDB) listens to the JMS queue.
Issue: After bringing everything up I am able to see 12 consumer count listening to the queue. But when I run the application which writes total of 13 messages in the queue only 7 of them are getting processed and 6 messages are always getting stuck in the queues. Interestingly this 7 messages are mostly getting processed by consumers with in the same windows box. And the selection of windows box is completely random. To confirm whether both the windows boxes are configured fine I tried to test them individually and it works fine individually, the problem happens only if I bring up both the boxes together.
Another observation is, if I bring down the non working domains, the pending messages start getting processed by the working domains. It gives an impression that the non working domains seem to lock the messages for processing but doesn't actually process it. Only when I bring down the domains it releases this lock.
Below is the JMS Life cycle events I captured from log files :
1. Start domain1 in WIN Box1
####<Nov 24, 2009 7:54:00 PM CST> <> <> <1259114040304> <783400> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session60.consumer62)> <> <>
####<Nov 24, 2009 7:54:00 PM CST> <> <> <1259114040309> <239900> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session63.consumer65)> <> <>
####<Nov 24, 2009 7:54:00 PM CST> <> <> <1259114040313> <2100> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session66.consumer68)> <> <>
2. Start domain2 in WIN Box 1
####<Nov 24, 2009 7:54:55 PM CST> <> <> <1259114095403> <38600> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session70.consumer72)> <> <>
####<Nov 24, 2009 7:54:55 PM CST> <> <> <1259114095407> <335400> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session73.consumer75)> <> <>
####<Nov 24, 2009 7:54:55 PM CST> <> <> <1259114095410> <997700> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session76.consumer78)> <> <>
3. Start domain1 in WIN Box 2
####<Nov 24, 2009 7:56:12 PM CST> <> <> <1259114172061> <316500> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session80.consumer82)> <> <>
####<Nov 24, 2009 7:56:12 PM CST> <> <> <1259114172066> <324300> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session83.consumer85)> <> <>
####<Nov 24, 2009 7:56:12 PM CST> <> <> <1259114172072> <757000> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session86.consumer88)> <> <>
4. Start domain2 in WIN Box 2
####<Nov 24, 2009 7:56:54 PM CST> <> <> <1259114214000> <142800> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session90.consumer92)> <> <>
####<Nov 24, 2009 7:56:54 PM CST> <> <> <1259114214005> <38200> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session93.consumer95)> <> <>
####<Nov 24, 2009 7:56:54 PM CST> <> <> <1259114214009> <22400> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session96.consumer98)> <> <>
At this point console shows 12 consumers listening the queue which is absolutely fine.
5. Run application
--Message Production
This produces 13 messages ( I have not pasted the complete log)
-- Message Consumption ( Only 7 messages are consumed)
####<Nov 24, 2009 8:00:05 PM CST> <> <> <1259114405953> <576400> <ID:<1047832.1259114392542.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session63.consumer65)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114392542</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:10 PM CST> <> <> <1259114410091> <310100> <ID:<1047832.1259114393181.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session66.consumer68)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114393181</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:10 PM CST> <> <> <1259114410130> <364600> <ID:<1047832.1259114400626.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session63.consumer65)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400626</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:10 PM CST> <> <> <1259114410134> <540700> <ID:<1047832.1259114393125.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session76.consumer78)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114393125</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:10 PM CST> <> <> <1259114410137> <632400> <ID:<1047832.1259114393128.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session73.consumer75)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114393128</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:12 PM CST> <> <> <1259114412026> <984200> <ID:<1047832.1259114393123.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session60.consumer62)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114393123</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:16 PM CST> <> <> <1259114416045> <41800> <ID:<1047832.1259114400005.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session70.consumer72)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400005</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
6. At this point Admin Consol shows 6 messages pending in the queue
and From consumer numbers it is evident that consumers in WIN Box2 domains have not processed any messages
Now if I stop server in WIN Box2 Domain1, WIN Box1 picks up the messages instantely
####<Nov 24, 2009 8:13:22 PM CST> <> <> <1259115202343> <897600> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session80.consumer82)> <> <>
####<Nov 24, 2009 8:13:22 PM CST> <> <> <1259115202346> <421900> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session83.consumer85)> <> <>
####<Nov 24, 2009 8:13:22 PM CST> <> <> <1259115202379> <582300> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session86.consumer88)> <> <>
####<Nov 24, 2009 8:13:24 PM CST> <> <> <1259115204888> <4600> <ID:<1047832.1259114400029.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session76.consumer78)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400029</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:13:32 PM CST> <> <> <1259115212820> <256700> <ID:<1047832.1259114400258.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session66.consumer68)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400258</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:13:43 PM CST> <> <> <1259115223579> <280600> <ID:<1047832.1259114400032.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session60.consumer62)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400032</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
7. At this poing Admin Consol shows 3 messages pending in the queue
Now if I stop server in WIN Box2 Domain2, the remaining 3 gets processed
####<Nov 24, 2009 8:16:08 PM CST> <> <> <1259115368998> <193400> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session90.consumer92)> <> <>
####<Nov 24, 2009 8:16:08 PM CST> <> <> <1259115369000> <885500> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session93.consumer95)> <> <>
####<Nov 24, 2009 8:16:09 PM CST> <> <> <1259115369002> <902500> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session96.consumer98)> <> <>
####<Nov 24, 2009 8:16:25 PM CST> <> <> <1259115385151> <123700> <ID:<1047832.1259114400324.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session70.consumer72)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400324</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:16:29 PM CST> <> <> <1259115389660> <836600> <ID:<1047832.1259114400535.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session63.consumer65)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400535</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:16:37 PM CST> <> <> <1259115397823> <610200> <ID:<1047832.1259114400432.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session73.consumer75)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400432</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
Kindly throw some light on this issue. Also do let me know if you need any more specific details about this.
Thanks,
Abhijeet
Edited by: user5240164 on Nov 25, 2009 11:35 AM
Edited by: user5240164 on Nov 28, 2009 1:21 PMHi,
The problem may be conflicting names in your configuration. To ensure proper operation of WL Security, JMS, JTA, and JDBC, make sure that (A) each domain has a unique name, and (B) no two WebLogic servers have the same name (even if in two different domains).
To ensure a more even distribution of messages, either simply send more than a token amount messages (eg, a hundred or so, rather than just 12), or configure a custom connection factory on the domain that hosts the queue, reduce MessagesMaximum on the connection factory to 1, and then modify each MDB to refer to the custom connection factory. The MessagesMaximum setting controls the number of messages that can be pre-pushed (pipelined) to a single asynchronous consumer.
For more information see "Best Practices for JMS Beginners and Advanced Users" in the latest versions of the JMS admin guide, and "JMS Performance & Tuning Check List" in the WebLogic Performance and Tuning guide.
Tom -
JMS messages stuck in queue in weblogic v923 cluster
Hi all,
Enviornment details:
In our pre-production weblogic v923 cluster enviornment with Oracle 10g database, there are 10 server boxes with 2 managed nodes running on each server box in linux enviornment.
There are 3 JMS queues and 1 error queue per managed node.
All the queues are durable and the persitence store is a database table. There is a single database table per single managed node.
High loads of JMS messages from client applications coming on each of the 3 JMS queue are processed by the cluster on daily basis.
The JMS messages that failed to be processed the very first time due to the application logic exceptions are put on error queue since the retry is set to zero on the non-error queue.
The application exceptions are logged to a database table for futher reference.
The falied messages from the error queue are read by spring based MDP and resent to the application for reprocessing.
If no application exceptions, the message will be persisted in a relational database; in the case of failures, MDP code puts the failed message back on to the error queue.
The auto_ack is true on the JMS message.
It is observed about less than 1% of the messages fail the original processing and are in error queue on daily basis.
99.9% of the time, the reprocessing of the messages from the error queue is a success.
Currently due to some obvious reasons, the system administrator stops a managed node when the disk is full and start a different managed node on the same server box. This approach will be stopped in the very near future so that all nodes on all servers are running at any point of time by making few enviornment changes.
Problem:
1a) Occasionally, it is observed that 1 - 4 JMS messages are stuck in the error queue. There are no signs of reprocessing as there are no application exceptions or the corresponing insert into the database. The JMS persistence store table still has the JMS message.
1b) It is observed that over time, the same JMS messages that were originally stuck on server 1's managed node A are no longer there but are found in a different node (say node B) on a different server box.
As far as I understand the JMS, a message could be in a Pending state if MDP (in my case) does not automatically send ACKNOWLEGMENT back due to an issue in the Spring Defaultlisternercontainer before calling MDP's onMessage(). I am not sure as what happens to that status over time).
It is also observed that the stuck messages content is not much different than the other messages that are processed fine.
Any pointers/input is highly appreciated.
Thanks in advance for your valuable input and time.Off the top of my head, I can only think of two reasons why a message might move from node to node:
(A) An application is dequeueing them and enquening them elsewhere.
(B) The application is using distributed queues, and the distributed queue has been configured to enable "queue forwarding". The forwarding feature automatically moves messages from distributed queue members that have no consumers to members that do have consumers. Queue forwarding is sometimes enabled as a work-around to better enable MDP support of WL distributed destinations, unlike WebLogic MDBs, MDPs have no container logic to ensure that all members of a distributed destination are serviced by a consumer under all circumstances...
Regards,
Tom -
I am on SOA SUITE 11.1.17
I have a jms queue set up and am able to successfully send messages to it using the SOA JMS Adapter. I am able to succesfully listen to a queue using a JMS adapter in a different project.
However, here when I try to "listen" to the queue using a java program, the message is "received" in my java program and processed as it should be. The java program is running in a while loop, so it stays running.
On the queue, when the java program reads it, the message is moved to the "Messages Pending" column. When I kill the java program, the message is returned to the "Messages Current"
Would anyone have any ideas about this? We have an existing java based program isn't properly "closing" the message.
I tried:
1. placing msg.acknowledge() in the code, since the documentation states that messages that are in "Messages Pending" have been read, but not acknowledged or committed -- but this does nothing.
2. Setting the expiration on the message to 1, but that is a hack. I would rather fix the problem than use a hack.
Anyone have any ideas what is wrong?
Thank you,
Stuart
import java.io.IOException;
import java.io.InputStreamReader;
import javax.jms.Connection;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.Session;
import javax.jms.TextMessage;
public class JMSListenAndRespond extends JMSQueueHandler implements MessageListener {
private Session jmsSession = null;
private MessageConsumer jmsMessageConsumer = null;
private Connection jmsConnection = null;
int counter = 1 ;
public static void main(String[] args) throws Exception {
InputStreamReader commandBuffer = null;
char command = '\0';
// first listen for any messages
JmsDataObject q = new JmsDataObject();
q.setJmsCF("jms/TestConnectionFactory");
q.setJmsQueue("jms/TestJMSQueue");
q.setServerPwd("welcome1");
q.setServerLogin("weblogic");
q.setUrl("t3://localhost:7101");
q.setJmsMsg("na -- we are listinging...");
JMSListenAndRespond lis = new JMSListenAndRespond();
lis.consumeMessage(q);
System.out.println("(type q to exit at any time...)\n");
try {
// Now just loop, waiting for messages until user types 'q' to quit
commandBuffer = new InputStreamReader(System.in);
while (!(command == 'q')) {
try {
command = (char)commandBuffer.read();
System.out.println("Command: "+command);
lis.stopListening();
} catch (IOException e) {
System.err.println("I/O Exception: ");
e.printStackTrace();
} //while
} finally {
lis.stopListening();
lis.stopListening();
public void stopListening() {
try {
if (jmsMessageConsumer != null) {
jmsMessageConsumer.close();
if (jmsSession != null) {
jmsSession.close();
if (jmsConnection != null) {
jmsConnection.close();
} catch (JMSException e) {
System.err.println("Couldn't properly close all resources.\n");
e.printStackTrace();
private void prepareToConsumeMessage() {
// create JMS connection, session, consumer and register listener - then start session
try {
// create connection
jmsConnection = getJmsConnectionFactory().createConnection();
// create session
jmsSession = jmsConnection.createSession(false, javax.jms.Session.AUTO_ACKNOWLEDGE);
jmsMessageConsumer = jmsSession.createConsumer(getJmsDestination());
// register a message listener (onMessage)
jmsMessageConsumer.setMessageListener(this);
// initiate connection - make sure to register your listener first or you might lose messages
jmsConnection.start();
System.out.println("\n...listening for message!\n");
} catch (JMSException e) {
System.err.println("Problem initializing JMS session");
e.printStackTrace();
System.exit(-1);
* onMessage
* The callback function invoked upon message reception
public void onMessage(Message msg) {
try {
String msgText;
if (msg instanceof TextMessage) {
msgText = ((TextMessage)msg).getText();
// this method expects a CSV string with the following makeup:
// firstName,lastName,gender,birthdate,address,city,zipcode,state,insurancePolicyId
String messageId = msg.getJMSMessageID();
// code to process goes here.....
} else {
// if message is not a TextMessage, attempt a conversion to string
System.out.println("\n** NEW MESSAGE (" + msg.getClass().getName() +
"):\n" +
msg.toString());
} catch (JMSException e) {
e.printStackTrace();
public void consumeMessage(JmsDataObject q ) {
initializeJndiProperties(q);
initializeJNDIandCF(q.getJmsQueue());
prepareToConsumeMessage();
import java.util.Properties;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.Destination;
import javax.naming.InitialContext;
import javax.naming.NamingException;
public abstract class JMSQueueHandler {
protected void initializeJNDIandCF(String destinationQueueName) {
// initialize JNDI context and lookup connection factory and destination
try {
// Initialize JNDI context
jndiContext = new InitialContext(jndiProperties);
} catch (NamingException e) {
e.printStackTrace();
try {
// lookup JMS connection factory in JNDI
jmsConnectionFactory =
(ConnectionFactory)jndiContext.lookup(jndiProperties.getProperty("factory.name"));
// lookup JMS destination in JNDI
jmsDestination = (Destination)jndiContext.lookup(destinationQueueName);
} catch (NamingException e) {
System.err.println("Problem during the JNDI lookup\n");
e.printStackTrace();
System.exit(-1);
public void initializeJndiProperties(JmsDataObject qdo ) {
// set JNDI properties
jndiProperties.put("java.naming.factory.initial","weblogic.jndi.WLInitialContextFactory");
jndiProperties.put("java.naming.provider.url", qdo.getUrl() );
// jndiProperties.put("java.naming.security.principal", NAMING_PRINCIPAL);
jndiProperties.put("java.naming.security.principal", qdo.getServerLogin() ) ;
// jndiProperties.put("java.naming.security.credentials", NAMING_CREDENTIAL);
jndiProperties.put("java.naming.security.credentials", qdo.getServerPwd() );
//jndiProperties.put("factory.name", JMS_CONNECTION_FACTORY);
jndiProperties.put("factory.name", qdo.getJmsCF() );
//protected static final String NAMING_PROVIDER_URL = "t3://localhost:7101";
//protected static final String NAMING_PRINCIPAL = "weblogic";
// protected static final String NAMING_CREDENTIAL = "welcome1";
// protected static final String JMS_CONNECTION_FACTORY = "jms/patientsJmsCF";
// protected static final String JMS_CONNECTION_FACTORY = "jms/TestConnectionFactory";
ConnectionFactory jmsConnectionFactory = null;
Destination jmsDestination = null;
InitialContext jndiContext = null;
Properties jndiProperties = new Properties();
public void setJmsConnectionFactory(ConnectionFactory jmsConnectionFactory) {
this.jmsConnectionFactory = jmsConnectionFactory;
public ConnectionFactory getJmsConnectionFactory() {
return jmsConnectionFactory;
public void setJmsDestination(Destination jmsDestination) {
this.jmsDestination = jmsDestination;
public Destination getJmsDestination() {
return jmsDestination;Thanks for the reply. There are no errors what so ever regarding MDB issues. Not in the weblogic / console logging, nor in the logging of the component itself.
The MDB itself is setup to have 16 concurrent beans per component deployment. The component is clustered over 4 server instances. So there are quite some concurrent MDBs running. However since the messages get stuck for weeks eventhough the component only has bursts of messages to process, one would think that the the stuck messages would be processed whenever the component has some spare time (which it has a lot between the bursts).
Therefor i doubt it has anything to do with too little MDBs running. But frankly I have no idea what's wrong. -
Weblogic Message Bridge Producing Duplicate JMS Messages
We have a message bridge that we use to integrate with TIBCO. The message bridge is deployed on a WL cluster. We see duplicate JMS messages produced by the bridge when it is deployed on the cluster. We don't see this behavior when only one ManagedServer in the cluster is active,.
TIBCO version: 4.4.3 V5
WLS: 10.3.3 with patch for bug BUG8732539 (BQC6)
java -version
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Oracle JRockit(R) (build R28.0.0-679-130297-1.6.0_17-20100312-2121-linux-x86_64, compiled mode)
We are also seeing a lot of exceptions related to
<May 27, 2011 4:11:57 PM PDT> <Warning> <Connector> <BEA-190032> << eis/jms/WLSConnectionFactoryJNDIXA > ResourceAllocationException thrown by resource adapter on call to ManagedConnectionFactory.createManagedConnection(): "javax.resource.ResourceException: Failed to start the connection ">
However the bridge is active. We are monitoring the message-out count on TIBCO and we observed that we see duplicates whenever there a message is re-delivered. We opened a case with Oracle support and they suggested patch 10258751 which didn't help. The next suggestion was a patch for "Bug 11852771 - [WLS10.3.2]MISSING AND DUPLICATE IN JMS QUEUE MESSAGE MONITORING" but I don't think it will help since our JMS consumers are getting duplicate messages and this is not just a UI issue.
Here is the message behavior when we send a single message:
Timelines
11:34:17 --> Message sent to TIBCO EMS queue
11:34:17 --> WL MS3 throws an javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMFAIL : Resource manager is unavailable exception
11:34:17 --> WL MS1 says it got the message and committed the transaction
11:34:17 --> Message is visible in the JMS queue in the admin console
11:34:17 --> TIBCO EMS has still not received an ACK from WL for the message
11:34:31 --> TIBCO EMS resends the same JMS message to WL
11:34:31 --> WL MS2 says it got the message and committed the transaction
11:34:31 --> WL MS1 throws an javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMFAIL : Resource manager is unavailable exception
11:34:31 --> Message is visible in the JMS queue in the admin console (At this point there are two messages in the JMS queue)
11:34:31 --> TIBCO EMS has still not received an ACK from WL for the message
11:34:54 --> TIBCO EMS resends the same JMS message to WL
11:34:31 --> WL MS4 says it got the message and committed the transaction
11:34:31 --> WL MS2 throws an javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMFAIL : Resource manager is unavailable exception
11:34:31 --> Message is visible in the JMS queue in the admin console (At this point there are three messages in the JMS queue)
11:34:31 --> TIBCO EMS receives an ACK from WL for the message
MS04
####<May 27, 2011 11:34:54 AM PDT> <Debug> <MessagingBridgeRuntime> <> <MS04> <[ACTIVE] ExecuteThread: '109' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-6A59E8FE76C4FC4CA149> <> <1306521294611> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) received message:
JMS Message Class: TextMessage
JMSMessageID: ID:MS01.4B044DDFEE1F1:25354AD
JMSCorrelationID: null
JMSDeliveryMode: PERSISTENT
JMSDestination: Queue[SourceQueue1]
JMSExpiration: 0
JMSPriority: 4
JMSRedelivered: true
JMSReplyTo: null
JMSTimestamp: 1306521256905 (Fri May 27 11:34:16 PDT 2011)
JMSType: null
Transaction Id: BEA1-6A59E8FE76C4FC4CA149
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
####<May 27, 2011 11:34:54 AM PDT> <Debug> <MessagingBridgeRuntime> <MS04> <MS04> <[ACTIVE] ExecuteThread: '109' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-6A59E8FE76C4FC4CA149> <> <1306521294611> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) successfully sent message:
JMS Message Class: TextMessage
Old JMS MessageID: ID:MS01.4B044DDFEE1F1:25354AD
New JMS MessageID: ID:<193654.1306521294611.0>
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
MS03
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257342> <BEA-000000> <Bridge MessageBridge1 Successfully got connection to the target destination>
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257343> <BEA-000000> <Bridge MessageBridge1: both source and target adapters support XA = true>
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257343> <BEA-000000> <Bridge MessageBridge1 supportsMDBTX = false supportsXA = true async = true>
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257343> <BEA-000000> <Bridge MessageBridge1 WorkMode = Exactly-once>
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntimeVerbose> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257343> <BEA-000000> <Bridge MessageBridge1 Entering processMessages() ------ >
####<May 27, 2011 11:34:17 AM PDT> <Error> <MessagingBridge> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-3AA29116A4CA622834EA> <> <1306521257343> <BEA-200015> <An error occurred in bridge "MessageBridge1" during the transfer of messages (java.lang.Exception: javax.resource.ResourceException: Failed to setup the Resource Adapter Connection for enlistment in the transaction, Pool = 'eis/jms/WLSConnectionFactoryJNDIXA', javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMFAIL : Resource manager is unavailable
javax.transaction.xa.XAException: Internal error: XAResource 'eis/jms/WLSConnectionFactoryJNDIXA' is unavailable
at weblogic.transaction.internal.XAResourceDescriptor.checkResource(XAResourceDescriptor.java:948)
at weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:634)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1227)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:507)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:434)
at weblogic.connector.transaction.outbound.XATxConnectionHandler.enListResource(XATxConnectionHandler.java:117)
at weblogic.connector.outbound.ConnectionWrapper.invoke(ConnectionWrapper.java:218)
at $Proxy62.receive(Unknown Source)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1431)
at weblogic.jms.bridge.internal.MessagingBridge.beginForwarding(MessagingBridge.java:1006)
at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1083)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
at weblogic.connector.transaction.outbound.XATxConnectionHandler.enListResource(XATxConnectionHandler.java:200)
at weblogic.connector.outbound.ConnectionWrapper.invoke(ConnectionWrapper.java:218)
at $Proxy62.receive(Unknown Source)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1431)
at $Proxy62.receive(Unknown Source)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1431)
at weblogic.jms.bridge.internal.MessagingBridge.beginForwarding(MessagingBridge.java:1006)
at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1083)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMFAIL : Resource manager is unavailable
javax.transaction.xa.XAException: Internal error: XAResource 'eis/jms/WLSConnectionFactoryJNDIXA' is unavailable
at weblogic.transaction.internal.XAResourceDescriptor.checkResource(XAResourceDescriptor.java:948)
at weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:634)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1227)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:507)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:434)
at weblogic.connector.transaction.outbound.XATxConnectionHandler.enListResource(XATxConnectionHandler.java:117)
at weblogic.connector.outbound.ConnectionWrapper.invoke(ConnectionWrapper.java:218)
at $Proxy62.receive(Unknown Source)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1431)
at weblogic.jms.bridge.internal.MessagingBridge.beginForwarding(MessagingBridge.java:1006)
at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1083)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1185)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:507)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:434)
at weblogic.connector.transaction.outbound.XATxConnectionHandler.enListResource(XATxConnectionHandler.java:117)
... 8 more
).>
####<May 27, 2011 11:34:17 AM PDT> <Warning> <MessagingBridge> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257344> <BEA-200026> <Bridge "MessageBridge1" encountered some problems in one of its adapters or underlying systems. It stopped transferring messages and will try to reconnect to the adapters shortly. (The exception caught was java.lang.Exception: MessagingBridgeException: Messaging bridge operation failed
.)>
MS02
####<May 27, 2011 11:34:31 AM PDT> <Debug> <MessagingBridgeRuntime> <MS02> <MS02> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-679AF074F381152F047C> <> <1306521271386> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) received message:
JMS Message Class: TextMessage
JMSMessageID: ID:MS01.4B044DDFEE1F1:25354AD
JMSCorrelationID: null
JMSDeliveryMode: PERSISTENT
JMSDestination: Queue[SourceQueue1]
JMSExpiration: 0
JMSPriority: 4
JMSRedelivered: true
JMSReplyTo: null
JMSTimestamp: 1306521256905 (Fri May 27 11:34:16 PDT 2011)
JMSType: null
Transaction Id: BEA1-679AF074F381152F047C
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
####<May 27, 2011 11:34:31 AM PDT> <Debug> <MessagingBridgeRuntime> <MS02> <MS02> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-679AF074F381152F047C> <> <1306521271392> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) successfully sent message:
JMS Message Class: TextMessage
Old JMS MessageID: ID:MS01.4B044DDFEE1F1:25354AD
New JMS MessageID: ID:<196075.1306521271386.0>
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
####<May 27, 2011 11:34:54 AM PDT> <Error> <MessagingBridge> <MS02> <MS02> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521294610> <BEA-200015> <An error occurred in bridge "MessageBridge1" during the transfer of messages (java.lang.Exception: weblogic.transaction.RollbackException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:495)
at weblogic.connector.transaction.outbound.XAWrapper.setTransactionTimeout(XAWrapper.java:457)
at weblogic.connector.transaction.outbound.RecoveryOnlyXAWrapper.setTransactionTimeout(RecoveryOnlyXAWrapper.java:34)
at weblogic.transaction.internal.XAResourceDescriptor.setXAResourceTransactionTimeoutIfAppropriate(XAResourceDescriptor.java:1422)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1223)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.lang.NullPointerException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:485)
... 14 more
at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:1871)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:330)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:225)
at weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManagerImpl.java:293)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1534)
at weblogic.jms.bridge.internal.MessagingBridge.beginForwarding(MessagingBridge.java:1006)
at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1083)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:495)
at weblogic.connector.transaction.outbound.XAWrapper.setTransactionTimeout(XAWrapper.java:457)
at weblogic.connector.transaction.outbound.RecoveryOnlyXAWrapper.setTransactionTimeout(RecoveryOnlyXAWrapper.java:34)
at weblogic.transaction.internal.XAResourceDescriptor.setXAResourceTransactionTimeoutIfAppropriate(XAResourceDescriptor.java:1422)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1223)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.lang.NullPointerException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:485)
... 14 more
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1185)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
... 2 more
).>
####<May 27, 2011 11:34:54 AM PDT> <Warning> <MessagingBridge> <MS02> <MS02> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521294610> <BEA-200026> <Bridge "MessageBridge1" encountered some problems in one of its adapters or underlying systems. It stopped transferring messages and will try to reconnect to the adapters shortly. (The exception caught was java.lang.Exception: MessagingBridgeException: Messaging bridge operation failed
MS01
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS01> <MS01> <[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-72BC818FE2BB8C2655C6> <> <1306521257536> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) received message:
JMS Message Class: TextMessage
JMSMessageID: ID:MS01.4B044DDFEE1F1:25354AD
JMSCorrelationID: null
JMSDeliveryMode: PERSISTENT
JMSDestination: Queue[SourceQueue1]
JMSExpiration: 0
JMSPriority: 4
JMSRedelivered: false
JMSReplyTo: null
JMSTimestamp: 1306521256905 (Fri May 27 11:34:16 PDT 2011)
JMSType: null
Transaction Id: BEA1-72BC818FE2BB8C2655C6
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS01> <MS01> <[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-72BC818FE2BB8C2655C6> <> <1306521257542> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) successfully sent message:
JMS Message Class: TextMessage
Old JMS MessageID: ID:MS01.4B044DDFEE1F1:25354AD
New JMS MessageID: ID:<195581.1306521257536.0>
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
####<May 27, 2011 11:34:31 AM PDT> <Error> <MessagingBridge> <MS01> <MS01> <[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521271384> <BEA-200015> <An error occurred in bridge "MessageBridge1" during the transfer of messages (java.lang.Exception: weblogic.transaction.RollbackException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:495)
at weblogic.connector.transaction.outbound.XAWrapper.setTransactionTimeout(XAWrapper.java:457)
at weblogic.connector.transaction.outbound.RecoveryOnlyXAWrapper.setTransactionTimeout(RecoveryOnlyXAWrapper.java:34)
at weblogic.transaction.internal.XAResourceDescriptor.setXAResourceTransactionTimeoutIfAppropriate(XAResourceDescriptor.java:1422)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1223)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.lang.NullPointerException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:485)
... 14 more
at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:1871)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:330)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:224)
at weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManagerImpl.java:293)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1534)
at weblogic.jms.bridge.internal.MessagingBridge.beginForwarding(MessagingBridge.java:1006)
at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1083)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:495)
at weblogic.connector.transaction.outbound.XAWrapper.setTransactionTimeout(XAWrapper.java:457)
at weblogic.connector.transaction.outbound.RecoveryOnlyXAWrapper.setTransactionTimeout(RecoveryOnlyXAWrapper.java:34)
at weblogic.transaction.internal.XAResourceDescriptor.setXAResourceTransactionTimeoutIfAppropriate(XAResourceDescriptor.java:1422)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1223)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.lang.NullPointerException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:485)
... 14 more
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1185)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
... 1 more
).>
####<May 27, 2011 11:34:31 AM PDT> <Warning> <MessagingBridge> <MS01> <MS01> <[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521271384> <BEA-200026> <Bridge "MessageBridge1" encountered some problems in one of its adapters or underlying systems. It stopped transferring messages and will try to reconnect to the adapters shortly. (The exception caught was java.lang.Exception: MessagingBridgeException: Messaging bridge operation failed
.)>@Tom, thanks for your hints and sorry for my delayed response. Here are my answers to your queries…
-- Does messaging work, without any exceptions, when only one bridge is active?
It works fine when only one node of the cluster is active. -- I'm not familiar with Tibco monitoring, but I suspect that the "message-out count on TIBCO" stat doesn't always imply that there's a duplicate message. It could mean that an initial message delivery attempt failed, it's transaction rolled back, and the message was redelivered. Redelivery after a failure is normal - the original delivered message, if one actually made it onto a WebLogic queue, should be discarded by WebLogic when its failed transaction eventually rolls back (before its ever handed to a consumer application).
You are absolutely right. The mismatch in count was due to redelivery attempts. We could see that as soon as we turned on trace in TIBCO-- The posted stack traces look like they may be JVM stdout/stderr scrapes, which are often automatically truncated, for example, see the "... 8 more" listed at the bottom of one of the stack traces below. There might be more information in the full stack trace for the "Resource manager is unavailable" exception. Full traces are logged in each WebLogic Server's log files.
I tried redirecting the output and even set "stdout Stack Trace Depth:" to -1 but I am not albe to see the complete trace.-- The setTransactionTimeout NullPointerException error message looks like it is collateral damage from the original reported "Resource manager is unavailable" exception that's thrown earlier during the enlistResource() step. If you are truly getting duplicates, then I think this is a clue that the bridge is somehow (stupidly) trying to proceed with forwarding the message despite the fact that there's no current transaction on the thread...
I have a case open with Oracle support and we have extensive debugging turned on. Here is what the engineer came back with. There is an issue with XA transactions in JCA adapter when it talks to third party providers like TIBCO. Since the resource names are not unique across the cluster, a XA transaction is started on multiple nodes in the cluster for the same message. All transactions abort but the message is persisted and TIBCO redelivers the message since the connection is severed when the transaction aborts. We were able to reproduce this scenario at will with our setup and Oracle was able to reproduce the issue too. -- Can you check the state of the message on the WebLogic Q? The fact that a message is shown on the WL console doesn't necessarily mean that the state of the message is visible (it could be waiting for the transaction to commit - something that apparently never happens). On the console: Services -> Messaging -> JMS Modules -> [module name] -> [destination name].
Our consumer was getting duplicate messages. This confirms that the console was not giving us incorrect information. The console showed that 2/3/4 messages were in "current" state when we expected only one message. -- Make sure that the bridge's Tibco destination is a Tibco queue not a Tibco topic (it's happened).
Yeah we made sure it was a Queue. -- Can you post your bridge and destination stanzas from your config.xml?
Will try to do this shortly but have to strip out lot of information. -- I vaguely recall that there was a problem with untyped connection factories in Tibco at one point (not sure). You might want to try changing the Tibco configuration to spin-up "XA Queue Connection Factory" instead of just an "XA Connection Factory" (plus modify the bridge configuration accordingly).
We tried this but it didn't eliminate this issue. One thing we did notice however was that the number of rollbacks were greatly reduced.
Here is the workaround that Oracle is suggesting: Create pinned destinations and assign it to a distributed queue. This will keep the transactions local and hence eliminate the duplicate transactions and rollback. We tried it and it eliminated the dups issue. However I discovered another bug during my testing. Our cluster messaging mode is "Unicast" and message forwarding from zero consumer queues to queues with consumers fails. We have four/five nodes in the cluster and two consumer threads. I observed that the zero consumer queues always forward to one queue in the cluster that has consumers. The forwarding stalls when I bring down the node that has consumers. Again, we were able to reproduce this issue at will and Oracle has acknowledged that it is a bug. They've reported that Unicast fails without any warning whereas Multicast fails with a warning in the log. We have observed this message forwarding stalling even with UDQs.
I have another case open for an issue where our message bridges that transfer messages from TIBCO to WLS stalls after sometime. The status is shown as active but it is not doing anything. The server instance shows stuck threads trying to roll back a message. TIBCO logs indicate that it is receiving a request to rollback a transaction but it is not finding the transaction to rollback. The only way to recover from this condition is to restart the cluster. I am fairly confident that he root case for this issue is again the XA transaction issue. I changed our UDQ to pinned destination in one environment and we are no longer seeing dups or bridge stalling there. Hopefully Oracle can resolve the message forwarding issue.Thanks again…. -
JMS message from one application to the another appl in the same container?
I have a question.
Can I send JMS message from one application to the another application in the same container?
If yes ...can you provide me info.
I want to know in regards of EJB2.0 and EJB3.0.
Thanks,
RahulSo the receiver application is a Message Driven Bean, right?
In your container, do you have a mean to ensure that it is correctly configured to listen the desired destination? Like an administrative console that would show the destinations, the number of pending messages, the number of active consummers...?
btw what is the destination type? A queue or a topic? -
Hello All,
I have two WLS domains. One WLS 10.3.X, another is WLS 8.1.X
Each domain have one queue each (e.g. 10.3.X => Q1 & 8.1.X => Q2) and the 10.3.X domain have been configured with a MessageBridge to deliver the messages from Q1 to Q2
Each domain have ConnectionFactories XA enabled and the Bridge adapter is also set to "eis.jms.WLSConnectionFactoryJNDIXA"
Now the Bridge is showing active, but if we put a test message in Q1, it is taking long to be delivered to Q2. There is no error in the logs (Debug Flgs have also been enabled).
Sometimes it is also not being delivered at all.
During this, the status of the message in the Q1 is showing "receive transaction".
So can anyone tell me :
1> what are the valid status of any message in wls jms queues and what are the meaning of them.
2> what is the lifecycle of a message.
Thanks in advancehttp://docs.oracle.com/cd/E24329_01/apirefs.1211/e24391/weblogic/jms/extensions/JMSMessageInfo.html#JMSMessageInfo(long,%20int,%20java.lang.String,%20long,%20java.lang.String,%20weblogic.jms.extensions.WLMessage,%20java.lang.String,%20boolean)
"visible - The message is visible on the destination and may be consumed.
send - The message is pending because it is associated with an in progress send operation.
receive - The message is pending because it has been received by a consumer but not yet acknowledged.
transaction - The message is associated with a transactional “send” or “receive” operation that is in progress.
ordered - The message belongs to a Unit-of-Order (UOO).
delayed - The message is pending because it has a scheduled delivery time or is otherwise delayed.
expired - The message has expired and is no longer available for consumption.
redelivery-count-exceeded - The message has exceeded the redelivery count set for the destination and is no longer available for consumption.
paused - The message is pending because of a destination pause operation .
sequenced - The message that is part of a SAF sequence.
unit-of-work-component - The message is part of an Unit-of-Work (UOW).
deleted - The message is marked for deletion."
You can enable message life cycle logging
- http://docs.oracle.com/cd/E24329_01/web.1211/e24385/troubleshoot.htm#i1128815
"The message life cycle is an external view of the events that a JMS message traverses through once it has been accepted by the JMS server, either through the JMS APIs or the JMS Message Management APIs. Message life cycle logging provides an administrator with easy access to information about the existence and status of JMS messages from the JMS server viewpoint. In particular, each message log contains information about basic life cycle events such as message production, consumption, and removal."
There is also a WLST script provided here (Re: Example using of getMessage(s) with JMS error queues? for browsing messages on a given destination. -
JMS messages not being redelivered correctly ..
Hi Folks
I am having this weird problem with JMS using workshop/weblogic 8.1 SP2. I am
using a JMS control from within my pageflow to post a JMS message to a predefined
JMS queue (created by workshop for every web project . Mine is called testweb.queue.AsyncDispatcher
as my web project is called testweb). Since pageflows cannot receive asysnchronus
callbacks , I have not implemented the callback handler for teh JMS control as
I just need to be able to post from my Pageflow . So far so good.
I have an MDB listening to testweb.queue.AsyncDispatcher and that basically tries
to open a URL connection to a thirdparty web site and if connect is not successful
it should keep trying after every 10 seconds. I have the MDB throw an EJBException
(to simulate message redelivery) . The problem is that the message is redelivered
exactly 8 times and then I get the below message on the console :-
<May 24, 2004 10:18:00 AM EDT> <Error> <WLW> <F1234PiyushK> <cgServer> <ExecuteThread:
'14' for queue: 'weblogic.kernel.Default'> <<anonymous>> <BEA1-001D75CA6E6EB041A149>
<000000> <onMessageException: java.lang.Exception: Unexpected message format>
I also have an MDB listening to the error queue testweb.queue.AsyncDispatcher_error
but its onMessage is never executed .
I am stuck on this for the last efw days so appreciate any help. Should I post
a message to the JMS queue directly from a session bean insetad of from a page
flow ?
my JMS queue config parameters are mentioned below :-
Time To Deliver Delivery 5000
Delivery Mode Override - No Override
Expiration Policy - Redirect
Redelivery Delay Override - 5000
Redelivery Delay Override - 10
Error Destination - testweb.queue.AsyncDispatcher_error
The other parameter values are the default values and I have not changed themI am having the same problem with a simple WLI timer channel that posts a string message called myMessage every 20 seconds (what I defined in the WLI console).
This works like a charm.
In my process.jpd file, the timer subscription is followed by a JMS control, with a sendTextMessage(myMessage) which sends the message to a queue
@jc:jms
send-type="queue" send-jndi-name="JmsTimerProcessWeb.queue.AsyncDispatcher"
receive-type="queue" receive-jndi-name="NightlySiteProcess.queue.AsyncDispatcher"
connection-factory-jndi-name="weblogic.jws.jms.QueueConnectionFactory"
All goes well while sending but the messages tell otherwise:
=== Timer sent message = This is the WLI message
=== JMS sent message = This is the WLI message
<Aug 24, 2005 1:19:30 PM MDT> <Error> <WLW> <000000> <onMessageException: java.lang.Exception: Unexpected message format>
What does this mean?
I tried to add a project with a Message-bean, I cannot get the string message. I tried with an ObjectMessage, the same scenario everytime.
Any suggestions? -
Need suggestion on implementing JMS message error recovery
Hi,
Our application has a JMS topic where we publish application events. Now, there can be scenarios where the consumers cannot process the message due to some infrastructure issues and would error out. We need a way so that those messages can be reprocessed again later. we are thinking of the following design for JMS message error recovery
1. Use a persistent TOPIC (this would ensure guaranteed delivery)
2. Configure a error destination on JMS topic e.g a jms queue
3. Have an error handling MDB listening to the Error destination. An error handling MDB would dequeue the errored messages from error destination and persist it to a Data base "error" table..
4. Provide a mechanism to republish those messages to topic (e.g a scheduler or admin ui or a command line utility) .. The messages would deleted from database "error" table and published to topic again....
A. Are there any issues with the above design which we need to handle?
B. Are there any additional steps required in a Cluster environment with a distributed topic and distribute error destination? (our error mdb will have one-copy-per-application setting)
B. From a performance angle, Is it OK to use persistent TOPIC ? Or will it better to persist the message to the db table and then publish it as a non persistent message ... ? (But i guess the performance should be more or less the same in both of these approaches)
C. Are there any other recommended design patterns for error recovery of JMS messages
Please advise.
Regards,
ArifThanks Tom !
We may not be able to go with the approach of delaying/pausing redelivery of the messsage because
1. Pausing entire MDB approach: Our MDB application consumes messages generated by different producers and our MDB needs to continue processing the messages even if messages corresponding to one producer is erroring out
2. Redelivery delay : This would only delay the retry of an errored message. But there would still be a problem if the message fails during all retries (i.e redelivery limit count). We don't want to lose this message. In our case, It is possible that a particular message cannot be processed due to unavailability of a third party system for hours or may be a day.
Basically, i am looking on approaches for a robust and performant error recovery/retry framework for our application (refer details in my first post on this thread) while fully making use of all features provided by middleware (WLS). Please advise.
Regards,
Arif -
Lost (and found) JMS - messages
Using WLS 6.1 and have experienced that JMS messages has not been delivered to
its consumer correctly. We are interested in any suggestions to what has happend,
what can be done to avoid the problem in the future, or perhaps how we can find
out more about the problem.
We have a message driven bean A which sends messages to the queue Q, of which
the message driven bean B is the only consumer.
What we recently have experienced is that some of the messages delivered by A
to Q never are delivered to B. But the messages seems to be persisted correctly,
because after a restart they reappear!
Following is a more detailed description of the scenario:
1. A message is sendt with DeliveryMode=PERSISTENT and TimeToLive=0. QueueSender.send
is invoked and returns normally.
2. The consumers never gets the message (B.onMessage i never invoked with this
message).
3. Point 1-2 is repeated for 13 different messages over a period when ~100 similar
messages are delivered normally.
4. The server is then restarted.
5. In the starup log it is indicated that 44 records are found in the JMSStore.
6. When the server restarts, the "lost" messages gets delivered to B.
After the server restart we have not experienced the problem again.
Logs does not show anything unusual about the previous restart.
Other details:
There is always exatly 1 instance of A running, and 5-20 instances of B. A implements
and uses a pool of javax.jms.XAQueueSession instances, which shares a single instance
of javax.jms.XAQueueConnection. The aknowledge-mode for B is "Auto-aknowledge".
We're using Weblogic Server 6.1.3.0 / Red Hat Linux 7.3.
Any help on this problem is appreciated!
I believe we've seen just the behaviour you describe on 6.1 SP2, but only on a
multi-cpu box (Solaris), and only under load. We can't reproduce the problem reliably,
and get no error messages/warnings anywhere, so very little to send to customer
support; by the time we notice this, the problem's long passed, so a thread dump's
of no use.
We're going to try and see how SP5 (which has loads of JMS fixes) pans out.
But if you find anything out, could you post here?
simon.
"Harald Stendal" <[email protected]> wrote:
>
>Using WLS 6.1 and have experienced that JMS messages has not been delivered
>to
>its consumer correctly. We are interested in any suggestions to what
>has happend,
>what can be done to avoid the problem in the future, or perhaps how we
>can find
>out more about the problem.
>
>We have a message driven bean A which sends messages to the queue Q,
>of which
>the message driven bean B is the only consumer.
>What we recently have experienced is that some of the messages delivered
>by A
>to Q never are delivered to B. But the messages seems to be persisted
>correctly,
>because after a restart they reappear!
>
>Following is a more detailed description of the scenario:
>
>1. A message is sendt with DeliveryMode=PERSISTENT and TimeToLive=0.
>QueueSender.send
>is invoked and returns normally.
>2. The consumers never gets the message (B.onMessage i never invoked
>with this
>message).
>3. Point 1-2 is repeated for 13 different messages over a period when
>~100 similar
>messages are delivered normally.
>4. The server is then restarted.
>5. In the starup log it is indicated that 44 records are found in the
>JMSStore.
>6. When the server restarts, the "lost" messages gets delivered to B.
>
>After the server restart we have not experienced the problem again.
>
>Logs does not show anything unusual about the previous restart.
>
>Other details:
>There is always exatly 1 instance of A running, and 5-20 instances of
>B. A implements
>and uses a pool of javax.jms.XAQueueSession instances, which shares a
>single instance
>of javax.jms.XAQueueConnection. The aknowledge-mode for B is "Auto-aknowledge".
>
>We're using Weblogic Server 6.1.3.0 / Red Hat Linux 7.3.
>
>Any help on this problem is appreciated!
>
-
Problem in sending JMS message on remote OC4J
I have two OC4J standalone (10.1.3.0.0 build 041119.0001.2385)
The containers work on Windows2000 in different machines connected by the LAN.
The First container has deployed application from example http://www.oracle.com/technology/tech/java/oc4j/1013/howtos/how-to-jca-intro/doc/how-to-jca-intro.html
The second container has j2ee application (Servlet) that sending JMS messages in the queue of the first container.
Code Servlet in second OC4J:
package mypackage2;
import javax.servlet. *;
import javax.servlet.http. *;
import java.io. PrintWriter;
import java.io. IOException;
import javax.jms. *;
import javax.naming. *;
import java.util. *;
public class Servlet1 extends HttpServlet
private static final String CONTENT_TYPE = "text/html;charset=windows-1251";
public void init (ServletConfig config) throws ServletException
super.init (config);
String QUEUE_NAME = "OracleASjms/MyQueue1";
String QUEUE_CONNECTION_FACTORY = "OracleASjms/MyQCF";
public void doGet (HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
response.setContentType (CONTENT_TYPE);
PrintWriter out = response.getWriter ();
out.println (" < html > ");
out.println (" < head > < title > Servlet1 < /title > < /head > ");
out.println (" < body > ");
try
Hashtable env = new Hashtable ();
env.put (Context. INITIAL_CONTEXT_FACTORY, "oracle.j2ee.rmi.RMIInitialContextFactory");
env.put (Context. SECURITY_PRINCIPAL, "admin");
env.put (Context. SECURITY_CREDENTIALS, "admin");
env.put (Context. PROVIDER_URL, "ormi://host_OC4J_1:23791/jcamdb");
env.put ("dedicated.rmicontext", "true");
InitialContext ic = new InitialContext (env);
QueueConnectionFactory connectionFactory = (QueueConnectionFactory)ic.lookup (QUEUE_CONNECTION_FACTORY);
QueueConnection connection = connectionFactory.createQueueConnection ();
connection.start ();
QueueSession queueSession =
connection.createQueueSession (false, Session.AUTO_ACKNOWLEDGE);
Queue queue = (Queue) ic.lookup (QUEUE_NAME);
ic.close ();
System.out.println (" Queue: " + queue);
QueueSender sender = queueSession.createSender (queue);
System.out.println (" creating Message: " + queue);
Message message = queueSession.createMessage ();
System.out.println (" Message created ");
message.setJMSType ("theMessage");
message.setLongProperty ("time", System.currentTimeMillis ());
message.setStringProperty ("id", "11111");
message.setStringProperty ("oamount", "55555");
message.setStringProperty ("message", "77777");
message.setStringProperty ("RECIPIENT", "MDB");
System.out.println (" Sending message... ");
sender.send (message);
System.out.println (" Message sent ");
sender.close ();
queueSession.close ();
connection.close ();
catch (Exception e)
System.out.println (" ** TEST FAILED ** < br > Exception: " + e);
out.println (e.toString ());
e.printStackTrace ();
out.println (" < p > The servlet has received a GET. This is the reply. < /p
");out.println (" < /body > < /html > ");
out.close ();
Error: This code send message in The First container, and should send in the second OC4J !!!!
Please answer :
As configure (what code it is necessary to write) servlet (any J2EE the application in OC4J) to use a path to OC4J JMS (remote OC4J JMS) through the Resource Adapter (using OracleASjms.rar). ???I have two OC4J standalone (10.1.3.0.0 build 041119.0001.2385)
The containers work on Windows2000 in different machines connected by the LAN.
The First container has deployed application from example http://www.oracle.com/technology/tech/java/oc4j/1013/howtos/how-to-jca-intro/doc/how-to-jca-intro.html
The second container has j2ee application (Servlet) that sending JMS messages in the queue of the first container.
Code Servlet in second OC4J:
package mypackage2;
import javax.servlet. *;
import javax.servlet.http. *;
import java.io. PrintWriter;
import java.io. IOException;
import javax.jms. *;
import javax.naming. *;
import java.util. *;
public class Servlet1 extends HttpServlet
private static final String CONTENT_TYPE = "text/html;charset=windows-1251";
public void init (ServletConfig config) throws ServletException
super.init (config);
String QUEUE_NAME = "OracleASjms/MyQueue1";
String QUEUE_CONNECTION_FACTORY = "OracleASjms/MyQCF";
public void doGet (HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
response.setContentType (CONTENT_TYPE);
PrintWriter out = response.getWriter ();
out.println (" < html > ");
out.println (" < head > < title > Servlet1 < /title > < /head > ");
out.println (" < body > ");
try
Hashtable env = new Hashtable ();
env.put (Context. INITIAL_CONTEXT_FACTORY, "oracle.j2ee.rmi.RMIInitialContextFactory");
env.put (Context. SECURITY_PRINCIPAL, "admin");
env.put (Context. SECURITY_CREDENTIALS, "admin");
env.put (Context. PROVIDER_URL, "ormi://host_OC4J_1:23791/jcamdb");
env.put ("dedicated.rmicontext", "true");
InitialContext ic = new InitialContext (env);
QueueConnectionFactory connectionFactory = (QueueConnectionFactory)ic.lookup (QUEUE_CONNECTION_FACTORY);
QueueConnection connection = connectionFactory.createQueueConnection ();
connection.start ();
QueueSession queueSession =
connection.createQueueSession (false, Session.AUTO_ACKNOWLEDGE);
Queue queue = (Queue) ic.lookup (QUEUE_NAME);
ic.close ();
System.out.println (" Queue: " + queue);
QueueSender sender = queueSession.createSender (queue);
System.out.println (" creating Message: " + queue);
Message message = queueSession.createMessage ();
System.out.println (" Message created ");
message.setJMSType ("theMessage");
message.setLongProperty ("time", System.currentTimeMillis ());
message.setStringProperty ("id", "11111");
message.setStringProperty ("oamount", "55555");
message.setStringProperty ("message", "77777");
message.setStringProperty ("RECIPIENT", "MDB");
System.out.println (" Sending message... ");
sender.send (message);
System.out.println (" Message sent ");
sender.close ();
queueSession.close ();
connection.close ();
catch (Exception e)
System.out.println (" ** TEST FAILED ** < br > Exception: " + e);
out.println (e.toString ());
e.printStackTrace ();
out.println (" < p > The servlet has received a GET. This is the reply. < /p
");out.println (" < /body > < /html > ");
out.close ();
Error: This code send message in The First container, and should send in the second OC4J !!!!
Please answer :
As configure (what code it is necessary to write) servlet (any J2EE the application in OC4J) to use a path to OC4J JMS (remote OC4J JMS) through the Resource Adapter (using OracleASjms.rar). ??? -
Specifying additional JMS message attributes in Sender Adapter
Hi,
Iam using sender JMS adapter and I have to use the option " Specify Additional JMS Message Properties" for adding addtionalJMS properties in XI message header.
Once I cselect the option in the below table Iam not able to add new lines to add addtional JMS Message Properties..But In receiver adapter I was able to add the new lines.
Is this option graded in SAP PI 7.3 or relatd to any authorization issue..?
Is anyone faced this issue before.. Please help me out
Thank youHi everybody
I'm starting a project where I have to take adptadores JMS and I'm in the same situation kirian, I am need to configure a JMS adapter with version of SAP PI 7.3 and I wonder if anyone can help me or give me manuals leagues where information comes related.
Someone can help me with this information
Regards -
How to stop and start MDBs to listen for JMS messages
Hello! This might be more of an architecture question rather than a technical questing, but I post it here, in the lack of a better place...
For several years I have been using Webster and Webster MQ to send messages back and forth to clients. The architecture for the middleware platform has been designed around the functionality of ListenerPorts. Listenerports is a Webster functionality that let you "stop" and start to listen to JMS messages for a given MDB either by admin console, or by JMX.
The arhitecture we used was something like this :
1.Client sends a persistent message to a queue. (order)
2. An MDB pics up the order message, calls a SessionBean that "fronts" the IIOP back end system.
This is quite "out of the box" architecture, but we used the Listenerports to solve a stability problem for our back end system.
The backEnd system was quite unstable, so the EJB threw an exception (typically because of timeout or connection refused) back to the MDB.
The Listenerport was "defined" to handle 3 errors before stopping, so the MDB resent the message twice before the listenerport automatically stopped. The nice thing about this feature was that the client could continue to send orders, and was not bothered when the back end system was down.
When the backEnd system was up and running, we could start the listenerport and it was all back to normal state. We even had a start script that checked the state of the listenerport every 30 minute, and started it automatically if it was stopped, making the admin task of starting the MDB unnecessary..
Now.. Why am I telling this story..
I would like to know how this could be done using "clean" J2EE technologies..That is..Not using any technology that is not portable..
I`m playing with glassfish at home, and it struck me that there is nothing in the J2EE spec that defines the functionality described above ? Am i missing something ?
I Do not want to stop My application because the back end system is down? That is the only way (i know of) that i can "stop" to listen for messages..
I can put all the messages on an error queue when the backbend system is down, but that would lead to more code just to handle the error messages. Maybe the easiest way to solve this is to "move" (programmatically) all messages from the error queue over to the "standard" queue every XX minute, but somehow that sounds "wrong"
Can anyone give me some advice to how this problem situation should be solved ? I`m interested in the solution both from an architectural perspective and from a more technical perspective..
Best regards
Hans-Jacob MelbyBreakpoints, whether normal or conditional, are just meant for debugging of your code. I had the sense from your question that the pausing you want to do is a part of normal operation of your code. I would NOT recommend using a breakpoint for that situation. It would bring up the block diagram showing the breakpoint when it occurs. A user besides the programmer would not know what to do in that case.
Yes, both the inner and outer loops would have shift registers.
Putting a case structure with a small while loop inside the "Pausing Case" is doable. It just depends on what you are doing or waiting for while the program operation is "paused". -
I am unable to send JMS message to application which resides in third party
I am able to produce the Message into JMS demoQueue which is available in SOA 10.1.3.3. And also I tried remote server which is available in my office it having the application SchoolMAX with JMS Configuration. So I need to produce the JMS message into SchoolMAX application. But I am unable to pass to the data’s to that SchoolMAX JMS.
for this I have configured the respective queues, queue factory etc in Enterprise Manager , but i am not able to access our application which is in remote server through third party in JDeveloper for BPEL Process.
please i am very new to this JDeveloper & BPEL Console , please help me out in resolving the issue ASAP. please we will be waiting for some reply.Hi Kathy,
Happy Holidays! Do you have an iPad with cellular service and a text-messaging plan?
Send messages with your iPhone, iPad, or iPod touch - Apple Support
http://support.apple.com/en-us/HT201287
What's the difference between SMS/MMS and iMessage?
SMS/MMS lets you send text messages and photos to other cellular phones or devices. iMessage lets you send text messages and photos to another iOS device over Wi-Fi, without worrying about text-messaging charges (cellular data charges may apply).
What do I need to use SMS/MMS and iMessage?
Here's what you need to use SMS/MMS and iMessage:
SMS/MMS: A text-messaging plan. Contact your carrier for more information.
iMessage: An iPhone, iPad, or iPod touch with iOS 5 or later.
How do I know if my message used SMS/MMS or iMessage?
You can tell which type of message you're sending by the color of the Send button and text bubble.
SMS/MMS: Send button and text bubbles are green.
iMessage: Send button and text bubbles are blue.
How can I link my iPhone number with Messages on my iPad, iPod touch, or Messages for Mac?
When you sign in to Messages with your Apple ID on your iPhone, you link your phone number to your Apple ID. If you then sign in to Messages with the same Apple ID on your iPad, iPod touch, or Mac, Messages will link that device to your phone number as well.
I hope this information helps ....
- Judy
Maybe you are looking for
-
Iphone 5 not syncing to itunes
When I try to sync the iPhone 5 with iTunes for the first time it says I need to update to 10.7. When I try updating to 10.7 it says, "This software requires Mac OS X version 10.6.8 or later." I have a 10.5.8. Do I need to update to 10.6.8 to sync t
-
Group by returning more than one column in a single query
Hello, Sorry to ask such a simple question I guess, but here is my problem. I'm building a query to get say the smallest user_id from employees in the company by department. The query goes like : select emp..department, min(emp.user_id) from employee
-
How to view instagram download in itunes
how to view instagram download in itunes
-
Eprint login error: Ajax submit failed: error = 403, Forbidden.
Why do I keep getting this error when trying to log in to eprint website to manage my printer: Ajax submit failed: error = 403, Forbidden.
-
Settings of Mail Host and Mail Port in transaction: SCOT for the node: SMTP
Hi, Could anyone please explain me the significance of Mail Host and Mail Port for the SMTP node in transaction: SCOT. It says Mail Server to which outbound mails can be passed. Can i specify my GMAIL ID in the field Mail Host and left blank the Mail