JMS Messages Stuck in Pending
Hi,
We have Weblogic JMS provider and we use spring JMS listener to read messages of the queues. Our Spring listeners are running inside a webMethods Integration Server container. For a few of the queues, we have been seeing messages that are stuck in the "Pending" state on the JMS queues. The state string inside webLogic is shown as "receive". We don't have any evidence from our logs that the message actually is received by the listener or by subsequent processes down stream. The messages gets cleared when the Weblogic server is restarted.
Has anyone faced a similar issue or any ideas as to what might be causing this stuck messages?
Our weblogic version is 9.2
Regards,
Aneel
Edited by: user3127831 on Mar 24, 2009 7:02 AM
Hi Aneel
Typically JMS messages get stuck in pending state between components because of XA issues - due to configuration on one side of the fence or another, transactions aren't committed successfully, so a JMS message is placed on a queue successfully, but is not available due to the transaction that created it not committing successfully.
What is creating the messages being placed on the queues? What's the difference between how you're creating messages for the queues you can read successfully and the ones that are not becoming available?
Similar Messages
-
JMS messages stuck as Pending messages
Hi.
I am trying to write an application to process huge amounts of data. I try to
process around 1,000, 000 messages [java objects] using a single queue and 8 MDB.
The strange thing is that in the end there are always 10 messages stuck in the
pending message category.
I can also observe that a weblogic thread is stuck. [can see it in the console]
The stuck thread current description is something like "Connectio27.Session28".
I have tried to use the QueueConnectionFactory from a EJB using resource reference
instead of fetching it directly. Even this has not helped. I guess the problem
is that while loading the messages onto the queues, one of the threads gets stuck
somewhere.
I am sing MDB's for processing. There doesnot seem to be any problems on the MDB
side.
If I reboot the server the 10 messages get processed successfully.
Any ideas will be highly appreciated.
-Sharma
if you waited for a minute or so between each threaddump, it definitly
sounds like an issue around the database query not returning or there is too
much data being returned.
sree
"Ghanshyam Sharma" <[email protected]> wrote in message
news:40e2d9d6$1@mktnews1...
>
> Hmm, I didnt wait too much between thread dumps. Probably a minute or two.
>
> Let me check the oracle stuff.
>
> Thanks a Lot.
>
> -Sharma
>
> "sree bodapati" <iwillreply@thenewsgroups> wrote:
> >Threaddumps dont seem to show much except for it looks like both execute
> >thread 13 and 12 are stuck reading from oracle , you may want to see
> >what
> >query is being sent to oracle by instrumenting you MDB code see that
> >query
> >properly returns. The threads are runnable but they dont seem to move
> >, what
> >was the duration between each thread dump ?
> >
> >/
> >sree
> >
> >
> >"Ghanshyam Sharma" <[email protected]> wrote in message
> >news:40e2ce39$1@mktnews1...
> >>
> >> Hi Tom.
> >>
> >> I am attaching a thread dump of the server on windows, when I have
> >20
> >messages
> >> stuck as pending with MDB's having not-supported transaction attribute.
> >>
> >> I hope this helps in debugging the problem. Please let me know if
> >something is
> >> needed.
> >>
> >> Thanks everyone.
> >> -Sharma
> >>
> >>
> >> Tom Barnes
>
><[email protected].bea.com>
> >> wrote:
> >> >As I wrote below, a thread-dump of the WL Server that hosts
> >> >the stuck MDB might help explain things.
> >> >
> >> >Sharma wrote:
> >> >
> >> >> Hm, this is confirmed. I pumped in one million messages a few time
> >> >with and without
> >> >> file stores to differe the performance, but every time I get a clean
> >> >slate :)
> >> >>
> >> >> Looks like having required transaction attribute somehow removes
> >this
> >> >problem.
> >> >>
> >> >>
> >> >> But still I do not understand why messages were getting stuck with
> >> >not-supported
> >> >> attribute. Tom is it possible to understand this. The
> >phenomenon/Reason
> >> >behind
> >> >> this may show up somewhere else :)
> >> >>
> >> >> -Sharma
> >> >>
> >> >> Tom Barnes
>
><[email protected].bea.com>
> >> >> wrote:
> >> >>
> >> >>>The default is no transactions. I also find it curious
> >> >>>that this helps. I wonder if tx timeouts are helping here...
> >> >>>
> >> >>>Sharma wrote:
> >> >>>
> >> >>>
> >> >>>>Surprisingly I could process all one million messages if I
configure
> >> >>>
> >> >>>the MDB's
> >> >>>
> >> >>>>as folowing.
> >> >>>>
> >> >>>>Default-transaction=required
> >> >>>>Transaction-typr=container
> >> >>>>
> >> >>>>When I was getting the problem, I had nothing for these settings.
> >> >I
> >> >>>
> >> >>>do not know
> >> >>>
> >> >>>>what is the default, probably Not-supported ???
> >> >>>>
> >> >>>>Let me try my testing a few times and than confirm to you guys.
> >> >>>>
> >> >>>>Tom Barnes
>
><[email protected].bea.com>
> >> >>>>wrote:
> >> >>>>
> >> >>>>
> >> >>>>>Since you narrowed the problem down to the MDB side, can
> >> >>>>>post a thread dump of the server when it is stuck? That
> >> >>>>>might help diagnose the problem.
> >> >>>>>
> >> >>>>>Ghanshyam Sharma wrote:
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>>Infact The thread which is stuck is not the loader/message
producer
> >> >>>>>
> >> >>>>>thread. It
> >> >>>>>
> >> >>>>>
> >> >>>>>>is a thread handling message driven beans. My producers end up
> >populating
> >> >>>>>
> >> >>>>>the
> >> >>>>>
> >> >>>>>
> >> >>>>>>JMS correctly.
> >> >>>>>>
> >> >>>>>>This is 81sp02.
> >> >>>>>>-Sharma
> >> >>>>>>
> >> >>>>>>"Ghanshyam Sharma" <[email protected]> wrote:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>>Hi.
> >> >>>>>>>
> >> >>>>>>>I am trying to write an application to process huge amounts
> >of
> >> >data.
> >> >>>>>>>I try to
> >> >>>>>>>process around 1,000, 000 messages [java objects] using a single
> >> >>>
> >> >>>queue
> >> >>>
> >> >>>>>>>and 8 MDB.
> >> >>>>>>>The strange thing is that in the end there are always 10
messages
> >> >>>
> >> >>>stuck
> >> >>>
> >> >>>>>>>in the
> >> >>>>>>>pending message category.
> >> >>>>>>>
> >> >>>>>>>I can also observe that a weblogic thread is stuck. [can see
> >it
> >> >in
> >> >>>>>
> >> >>>>>the
> >> >>>>>
> >> >>>>>
> >> >>>>>>>console]
> >> >>>>>>>The stuck thread current description is something like
> >"Connectio27.Session28".
> >> >>>>>>>
> >> >>>>>>>I have tried to use the QueueConnectionFactory from a EJB using
> >> >resource
> >> >>>>>>>reference
> >> >>>>>>>instead of fetching it directly. Even this has not helped. I
> >guess
> >> >>>>>
> >> >>>>>the
> >> >>>>>
> >> >>>>>
> >> >>>>>>>problem
> >> >>>>>>>is that while loading the messages onto the queues, one of the
> >> >threads
> >> >>>>>>>gets stuck
> >> >>>>>>>somewhere.
> >> >>>>>>>
> >> >>>>>>>I am sing MDB's for processing. There doesnot seem to be any
> >problems
> >> >>>>>>>on the MDB
> >> >>>>>>>side.
> >> >>>>>>>
> >> >>>>>>>If I reboot the server the 10 messages get processed
successfully.
> >> >>>>>>>
> >> >>>>>>>Any ideas will be highly appreciated.
> >> >>>>>>>-Sharma
> >> >>>>>>
> >> >>>>>>
> >> >>
> >> >
> >>
> >
> >
>
-
JMS messages stuck in queue in weblogic v923 cluster
Hi all,
Enviornment details:
In our pre-production weblogic v923 cluster enviornment with Oracle 10g database, there are 10 server boxes with 2 managed nodes running on each server box in linux enviornment.
There are 3 JMS queues and 1 error queue per managed node.
All the queues are durable and the persitence store is a database table. There is a single database table per single managed node.
High loads of JMS messages from client applications coming on each of the 3 JMS queue are processed by the cluster on daily basis.
The JMS messages that failed to be processed the very first time due to the application logic exceptions are put on error queue since the retry is set to zero on the non-error queue.
The application exceptions are logged to a database table for futher reference.
The falied messages from the error queue are read by spring based MDP and resent to the application for reprocessing.
If no application exceptions, the message will be persisted in a relational database; in the case of failures, MDP code puts the failed message back on to the error queue.
The auto_ack is true on the JMS message.
It is observed about less than 1% of the messages fail the original processing and are in error queue on daily basis.
99.9% of the time, the reprocessing of the messages from the error queue is a success.
Currently due to some obvious reasons, the system administrator stops a managed node when the disk is full and start a different managed node on the same server box. This approach will be stopped in the very near future so that all nodes on all servers are running at any point of time by making few enviornment changes.
Problem:
1a) Occasionally, it is observed that 1 - 4 JMS messages are stuck in the error queue. There are no signs of reprocessing as there are no application exceptions or the corresponing insert into the database. The JMS persistence store table still has the JMS message.
1b) It is observed that over time, the same JMS messages that were originally stuck on server 1's managed node A are no longer there but are found in a different node (say node B) on a different server box.
As far as I understand the JMS, a message could be in a Pending state if MDP (in my case) does not automatically send ACKNOWLEGMENT back due to an issue in the Spring Defaultlisternercontainer before calling MDP's onMessage(). I am not sure as what happens to that status over time).
It is also observed that the stuck messages content is not much different than the other messages that are processed fine.
Any pointers/input is highly appreciated.
Thanks in advance for your valuable input and time.Off the top of my head, I can only think of two reasons why a message might move from node to node:
(A) An application is dequeueing them and enquening them elsewhere.
(B) The application is using distributed queues, and the distributed queue has been configured to enable "queue forwarding". The forwarding feature automatically moves messages from distributed queue members that have no consumers to members that do have consumers. Queue forwarding is sometimes enabled as a work-around to better enable MDP support of WL distributed destinations, unlike WebLogic MDBs, MDPs have no container logic to ensure that all members of a distributed destination are serviced by a consumer under all circumstances...
Regards,
Tom -
JDBC Driver: Messages stuck in processing started at rwb
Hi,
I have ABAP Proxy (7.1 EHP1) -> JDBC (MS SQL Server) scenario.
I'm trying to do INSERT_UPDATE. The interfases works fine for the first 5 messages. If I check the DB I see rows inserted correctly, but the first five messages stucks in processing started, and the others JDBC messages stuck in Pending to be delivered The interfase is developed as asynchron, i'm not affraid in the response.
I tryed to do just an INSERT, send just one message, or change the isolation level. But the messages continues stucking in processing started status.
One of the lasts possibilities is transform the interface to sync or increase the max number of threads of queue "JDBC_http://sap.com/xi/XI/SystemRecv" via ConfigTool. But I can't stop the j2ee stack until next weekend.
The audit Log:
- The message was successfully received by the messaging system. Protocol: XI URL: http://sapdev:51000/MessagingSystem/receive/AFW/XI Credential (User): PIISUSER
- Using connection JDBC_http://sap.com/xi/XI/System. Trying to put the message into the receive queue.
- Message successfully put into the queue.
- The message was successfully retrieved from the receive queue.
- The message status was set to DLNG.
- Delivering to channel: CC_C0400_IM01_MES_INTEG
- JDBC Adapter Receiver processing started, required QoS ExactlyOnce
- JDBC Adapter Receiver Channel CC_C0400_IM01_MES_INTEG: processing started; party / service XXXXXXXXX_DES.
- INSERT INTO XXXXXXXXXX_TST (CodArticulo, Descripcion, EAN13, Vidautil, bolsasporestuche, estuchesporcaja, estuchesporpalet, gxu, UNISTK) VALUES (XXXXXX, XXXXXXXXX, , 270 , 1, 0 , 0.000 , 1000 , KG)
Maybe is a problem in DB side, but the experience tell me that is more inestable a SAP PI than MS SQL Server
Any Idea?
P.S. In the same PI is coexisting diferents JDBC interfaces that works fine. I have tried to use their channels but still stucking.
And this is not my first PI :PHi Xavier,
have a look at the documentation of this option:
db.autoCommit=NO|YES
Use this parameter to deactivate the logical unit of work required by the JDBC adapter to guarantee that the data in the database is consistent. This option is required for JDBC drivers that do not support transactions. To avoid data inconsistencies in the database when the logical unit of work is deactivated, ensure that multiple database transactions cannot access the database simultaneously.
The default value is NO.
Do not set this parameter to YES if the JDBC driver supports transactions, that is, if the system does not display a corresponding error message during normal operation.
Netweaver Documentation
I'm sure MS SQL Server JDBC Driver is able to support transactions. By default the whole message will be treated as one transaction against the database.
Regards
Sven -
Messages stuck in "Messages Pending" column of a JMS queue.
Hi
Is there any way to retrive the messages which are stuck in "Messages Pending" column of a JMS queue.
weblogic 8.1 SP5 version.
Thanksthere is some possibility to prevent the Pending Message or to automatically purge them on WL 9.x or WL 10.x?WebLogic JMS provides "message management" console, scripting, and mbean features to view and manipulate messages in a destination. You can search for "message management" in this newsgroups for more information - here's a good post: Re: WLST Script Help It also provides features for automatically moving problem messages to an error destination, expiring old messages, and/or deleting messages thave have exceeded a configured redelivery limit. I suggest reading through our feature list in the documentation, see "Value-Added WebLogic Server JMS Features" in the [ Information Roadmap | http://download.oracle.com/docs/cd/E15523_01/web.1111/e14529/messaging.htm#sthref23 ].
Note that it's usually best to try and understand why messages are marked pending and fix the underlying reason instead of purging such messages. Messages are normally pending for a valid reason, and most use cases can't tolerate arbitrarily losing a message.
Tom -
JMS Messages getting stuck in queues
Hi,
I am facing this peculiar issue with queue messages not getting picked up.
Application Architecture: There is 1 JMS queue ( which resides in WLDomain1 with in a Unix Box) and there are 2 windows boxes having 2 weblogic domains each with 3 instances of MDB deployed on each of the domains. So total of 12 consumers (2 Box * 2 Domain * 3 Instances of MDB) listens to the JMS queue.
Issue: After bringing everything up I am able to see 12 consumer count listening to the queue. But when I run the application which writes total of 13 messages in the queue only 7 of them are getting processed and 6 messages are always getting stuck in the queues. Interestingly this 7 messages are mostly getting processed by consumers with in the same windows box. And the selection of windows box is completely random. To confirm whether both the windows boxes are configured fine I tried to test them individually and it works fine individually, the problem happens only if I bring up both the boxes together.
Another observation is, if I bring down the non working domains, the pending messages start getting processed by the working domains. It gives an impression that the non working domains seem to lock the messages for processing but doesn't actually process it. Only when I bring down the domains it releases this lock.
Below is the JMS Life cycle events I captured from log files :
1. Start domain1 in WIN Box1
####<Nov 24, 2009 7:54:00 PM CST> <> <> <1259114040304> <783400> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session60.consumer62)> <> <>
####<Nov 24, 2009 7:54:00 PM CST> <> <> <1259114040309> <239900> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session63.consumer65)> <> <>
####<Nov 24, 2009 7:54:00 PM CST> <> <> <1259114040313> <2100> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session66.consumer68)> <> <>
2. Start domain2 in WIN Box 1
####<Nov 24, 2009 7:54:55 PM CST> <> <> <1259114095403> <38600> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session70.consumer72)> <> <>
####<Nov 24, 2009 7:54:55 PM CST> <> <> <1259114095407> <335400> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session73.consumer75)> <> <>
####<Nov 24, 2009 7:54:55 PM CST> <> <> <1259114095410> <997700> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session76.consumer78)> <> <>
3. Start domain1 in WIN Box 2
####<Nov 24, 2009 7:56:12 PM CST> <> <> <1259114172061> <316500> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session80.consumer82)> <> <>
####<Nov 24, 2009 7:56:12 PM CST> <> <> <1259114172066> <324300> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session83.consumer85)> <> <>
####<Nov 24, 2009 7:56:12 PM CST> <> <> <1259114172072> <757000> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session86.consumer88)> <> <>
4. Start domain2 in WIN Box 2
####<Nov 24, 2009 7:56:54 PM CST> <> <> <1259114214000> <142800> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session90.consumer92)> <> <>
####<Nov 24, 2009 7:56:54 PM CST> <> <> <1259114214005> <38200> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session93.consumer95)> <> <>
####<Nov 24, 2009 7:56:54 PM CST> <> <> <1259114214009> <22400> <> <> <JMSModule!Queue> <ConsumerCreate> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session96.consumer98)> <> <>
At this point console shows 12 consumers listening the queue which is absolutely fine.
5. Run application
--Message Production
This produces 13 messages ( I have not pasted the complete log)
-- Message Consumption ( Only 7 messages are consumed)
####<Nov 24, 2009 8:00:05 PM CST> <> <> <1259114405953> <576400> <ID:<1047832.1259114392542.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session63.consumer65)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114392542</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:10 PM CST> <> <> <1259114410091> <310100> <ID:<1047832.1259114393181.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session66.consumer68)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114393181</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:10 PM CST> <> <> <1259114410130> <364600> <ID:<1047832.1259114400626.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session63.consumer65)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400626</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:10 PM CST> <> <> <1259114410134> <540700> <ID:<1047832.1259114393125.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session76.consumer78)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114393125</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:10 PM CST> <> <> <1259114410137> <632400> <ID:<1047832.1259114393128.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session73.consumer75)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114393128</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:12 PM CST> <> <> <1259114412026> <984200> <ID:<1047832.1259114393123.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session60.consumer62)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114393123</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:00:16 PM CST> <> <> <1259114416045> <41800> <ID:<1047832.1259114400005.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session70.consumer72)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400005</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
6. At this point Admin Consol shows 6 messages pending in the queue
and From consumer numbers it is evident that consumers in WIN Box2 domains have not processed any messages
Now if I stop server in WIN Box2 Domain1, WIN Box1 picks up the messages instantely
####<Nov 24, 2009 8:13:22 PM CST> <> <> <1259115202343> <897600> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session80.consumer82)> <> <>
####<Nov 24, 2009 8:13:22 PM CST> <> <> <1259115202346> <421900> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session83.consumer85)> <> <>
####<Nov 24, 2009 8:13:22 PM CST> <> <> <1259115202379> <582300> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection79.session86.consumer88)> <> <>
####<Nov 24, 2009 8:13:24 PM CST> <> <> <1259115204888> <4600> <ID:<1047832.1259114400029.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session76.consumer78)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400029</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:13:32 PM CST> <> <> <1259115212820> <256700> <ID:<1047832.1259114400258.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session66.consumer68)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400258</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:13:43 PM CST> <> <> <1259115223579> <280600> <ID:<1047832.1259114400032.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session60.consumer62)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400032</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
7. At this poing Admin Consol shows 3 messages pending in the queue
Now if I stop server in WIN Box2 Domain2, the remaining 3 gets processed
####<Nov 24, 2009 8:16:08 PM CST> <> <> <1259115368998> <193400> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session90.consumer92)> <> <>
####<Nov 24, 2009 8:16:08 PM CST> <> <> <1259115369000> <885500> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session93.consumer95)> <> <>
####<Nov 24, 2009 8:16:09 PM CST> <> <> <1259115369002> <902500> <> <> <JMSModule!Queue> <ConsumerDestroy> <<anonymous>> <MC:CA(/<IP_BOX2>):OAMI(server.jms.connection89.session96.consumer98)> <> <>
####<Nov 24, 2009 8:16:25 PM CST> <> <> <1259115385151> <123700> <ID:<1047832.1259114400324.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session70.consumer72)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400324</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:16:29 PM CST> <> <> <1259115389660> <836600> <ID:<1047832.1259114400535.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection59.session63.consumer65)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400535</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
####<Nov 24, 2009 8:16:37 PM CST> <> <> <1259115397823> <610200> <ID:<1047832.1259114400432.0>> <> <JMSModule!Queue> <Consumed> <<anonymous>> <MC:CA(/<IP_BOX1>):OAMI(server.jms.connection69.session73.consumer75)> <<?xml version="1.0" encoding="UTF-8"?>
<mes:WLJMSMessage xmlns:mes="http://www.bea.com/WLS/JMS/Message"><mes:Header><mes:JMSTimestamp>1259114400432</mes:JMSTimestamp><mes:Properties/></mes:Header></mes:WLJMSMessage>> <>
Kindly throw some light on this issue. Also do let me know if you need any more specific details about this.
Thanks,
Abhijeet
Edited by: user5240164 on Nov 25, 2009 11:35 AM
Edited by: user5240164 on Nov 28, 2009 1:21 PMHi,
The problem may be conflicting names in your configuration. To ensure proper operation of WL Security, JMS, JTA, and JDBC, make sure that (A) each domain has a unique name, and (B) no two WebLogic servers have the same name (even if in two different domains).
To ensure a more even distribution of messages, either simply send more than a token amount messages (eg, a hundred or so, rather than just 12), or configure a custom connection factory on the domain that hosts the queue, reduce MessagesMaximum on the connection factory to 1, and then modify each MDB to refer to the custom connection factory. The MessagesMaximum setting controls the number of messages that can be pre-pushed (pipelined) to a single asynchronous consumer.
For more information see "Best Practices for JMS Beginners and Advanced Users" in the latest versions of the JMS admin guide, and "JMS Performance & Tuning Check List" in the WebLogic Performance and Tuning guide.
Tom -
I am on SOA SUITE 11.1.17
I have a jms queue set up and am able to successfully send messages to it using the SOA JMS Adapter. I am able to succesfully listen to a queue using a JMS adapter in a different project.
However, here when I try to "listen" to the queue using a java program, the message is "received" in my java program and processed as it should be. The java program is running in a while loop, so it stays running.
On the queue, when the java program reads it, the message is moved to the "Messages Pending" column. When I kill the java program, the message is returned to the "Messages Current"
Would anyone have any ideas about this? We have an existing java based program isn't properly "closing" the message.
I tried:
1. placing msg.acknowledge() in the code, since the documentation states that messages that are in "Messages Pending" have been read, but not acknowledged or committed -- but this does nothing.
2. Setting the expiration on the message to 1, but that is a hack. I would rather fix the problem than use a hack.
Anyone have any ideas what is wrong?
Thank you,
Stuart
import java.io.IOException;
import java.io.InputStreamReader;
import javax.jms.Connection;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.Session;
import javax.jms.TextMessage;
public class JMSListenAndRespond extends JMSQueueHandler implements MessageListener {
private Session jmsSession = null;
private MessageConsumer jmsMessageConsumer = null;
private Connection jmsConnection = null;
int counter = 1 ;
public static void main(String[] args) throws Exception {
InputStreamReader commandBuffer = null;
char command = '\0';
// first listen for any messages
JmsDataObject q = new JmsDataObject();
q.setJmsCF("jms/TestConnectionFactory");
q.setJmsQueue("jms/TestJMSQueue");
q.setServerPwd("welcome1");
q.setServerLogin("weblogic");
q.setUrl("t3://localhost:7101");
q.setJmsMsg("na -- we are listinging...");
JMSListenAndRespond lis = new JMSListenAndRespond();
lis.consumeMessage(q);
System.out.println("(type q to exit at any time...)\n");
try {
// Now just loop, waiting for messages until user types 'q' to quit
commandBuffer = new InputStreamReader(System.in);
while (!(command == 'q')) {
try {
command = (char)commandBuffer.read();
System.out.println("Command: "+command);
lis.stopListening();
} catch (IOException e) {
System.err.println("I/O Exception: ");
e.printStackTrace();
} //while
} finally {
lis.stopListening();
lis.stopListening();
public void stopListening() {
try {
if (jmsMessageConsumer != null) {
jmsMessageConsumer.close();
if (jmsSession != null) {
jmsSession.close();
if (jmsConnection != null) {
jmsConnection.close();
} catch (JMSException e) {
System.err.println("Couldn't properly close all resources.\n");
e.printStackTrace();
private void prepareToConsumeMessage() {
// create JMS connection, session, consumer and register listener - then start session
try {
// create connection
jmsConnection = getJmsConnectionFactory().createConnection();
// create session
jmsSession = jmsConnection.createSession(false, javax.jms.Session.AUTO_ACKNOWLEDGE);
jmsMessageConsumer = jmsSession.createConsumer(getJmsDestination());
// register a message listener (onMessage)
jmsMessageConsumer.setMessageListener(this);
// initiate connection - make sure to register your listener first or you might lose messages
jmsConnection.start();
System.out.println("\n...listening for message!\n");
} catch (JMSException e) {
System.err.println("Problem initializing JMS session");
e.printStackTrace();
System.exit(-1);
* onMessage
* The callback function invoked upon message reception
public void onMessage(Message msg) {
try {
String msgText;
if (msg instanceof TextMessage) {
msgText = ((TextMessage)msg).getText();
// this method expects a CSV string with the following makeup:
// firstName,lastName,gender,birthdate,address,city,zipcode,state,insurancePolicyId
String messageId = msg.getJMSMessageID();
// code to process goes here.....
} else {
// if message is not a TextMessage, attempt a conversion to string
System.out.println("\n** NEW MESSAGE (" + msg.getClass().getName() +
"):\n" +
msg.toString());
} catch (JMSException e) {
e.printStackTrace();
public void consumeMessage(JmsDataObject q ) {
initializeJndiProperties(q);
initializeJNDIandCF(q.getJmsQueue());
prepareToConsumeMessage();
import java.util.Properties;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.Destination;
import javax.naming.InitialContext;
import javax.naming.NamingException;
public abstract class JMSQueueHandler {
protected void initializeJNDIandCF(String destinationQueueName) {
// initialize JNDI context and lookup connection factory and destination
try {
// Initialize JNDI context
jndiContext = new InitialContext(jndiProperties);
} catch (NamingException e) {
e.printStackTrace();
try {
// lookup JMS connection factory in JNDI
jmsConnectionFactory =
(ConnectionFactory)jndiContext.lookup(jndiProperties.getProperty("factory.name"));
// lookup JMS destination in JNDI
jmsDestination = (Destination)jndiContext.lookup(destinationQueueName);
} catch (NamingException e) {
System.err.println("Problem during the JNDI lookup\n");
e.printStackTrace();
System.exit(-1);
public void initializeJndiProperties(JmsDataObject qdo ) {
// set JNDI properties
jndiProperties.put("java.naming.factory.initial","weblogic.jndi.WLInitialContextFactory");
jndiProperties.put("java.naming.provider.url", qdo.getUrl() );
// jndiProperties.put("java.naming.security.principal", NAMING_PRINCIPAL);
jndiProperties.put("java.naming.security.principal", qdo.getServerLogin() ) ;
// jndiProperties.put("java.naming.security.credentials", NAMING_CREDENTIAL);
jndiProperties.put("java.naming.security.credentials", qdo.getServerPwd() );
//jndiProperties.put("factory.name", JMS_CONNECTION_FACTORY);
jndiProperties.put("factory.name", qdo.getJmsCF() );
//protected static final String NAMING_PROVIDER_URL = "t3://localhost:7101";
//protected static final String NAMING_PRINCIPAL = "weblogic";
// protected static final String NAMING_CREDENTIAL = "welcome1";
// protected static final String JMS_CONNECTION_FACTORY = "jms/patientsJmsCF";
// protected static final String JMS_CONNECTION_FACTORY = "jms/TestConnectionFactory";
ConnectionFactory jmsConnectionFactory = null;
Destination jmsDestination = null;
InitialContext jndiContext = null;
Properties jndiProperties = new Properties();
public void setJmsConnectionFactory(ConnectionFactory jmsConnectionFactory) {
this.jmsConnectionFactory = jmsConnectionFactory;
public ConnectionFactory getJmsConnectionFactory() {
return jmsConnectionFactory;
public void setJmsDestination(Destination jmsDestination) {
this.jmsDestination = jmsDestination;
public Destination getJmsDestination() {
return jmsDestination;Thanks for the reply. There are no errors what so ever regarding MDB issues. Not in the weblogic / console logging, nor in the logging of the component itself.
The MDB itself is setup to have 16 concurrent beans per component deployment. The component is clustered over 4 server instances. So there are quite some concurrent MDBs running. However since the messages get stuck for weeks eventhough the component only has bursts of messages to process, one would think that the the stuck messages would be processed whenever the component has some spare time (which it has a lot between the bursts).
Therefor i doubt it has anything to do with too little MDBs running. But frankly I have no idea what's wrong. -
I am running PI7.0 SP10 on a MCOD system running ECC6.0 and EP.
We had an issue last week on our Production system where the Java Stack went down, and when the system came back up, we cleared the issues in SMQ1, SMQ2 and SM58 and now we have over two hundred messages stuck in the java queues.
There is nothing locked in SM13, in sxmb_moni all the messgaes are successfully processed... In the RWB the status of the messages are "To be delivered".
I have tried reactivating the adapters but this has not worked.
The queues can be seen in the http://host.fqdn:5<SYSNO>00/MessagingSystem/monitor/systemStatus.jsp
specifically : File_http://sap.com/xi/XI/SystemRecv
and
SOAP_http://sap.com/xi/XI/SystemRecv
where the "isStarted" is set to false, where all the others are true...
How can I go about starting these and processing the backlog of messages??
I have searched OSS, SDN and SAPFans but had not luck!
I have subsequently done a full cache refresh and CPACache refresh but this has not helped.
There are no locked users (XI/PI users) holding this up!Ok,
It seems to be ok, the return can be http 200 or http 500.
The Soap and File are destinations, ok.
The messages are with status "delivering" on rwb... really, increasing the number of threads will not solve this...
I don't have access to visual admin (only basis team can access it here). I thought it could be something wrong with destination (but you already checked it)... so, the only thing can be done is to find in visual admin somewhere about system properties, conection parameters searching for something wrong there... here are our parameters if you would like to find something (sorry for the size of message)...
AFW.Call.maxConsumers 10
AFW.Recv.maxConsumers 5
AFW.Rqst.maxConsumers 10
AFW.Send.maxConsumers 5
BC_http://sap.com/xi/XI/System.Call.maxConsumers 20
BC_http://sap.com/xi/XI/System.Recv.maxConsumers 10
BC_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
BC_http://sap.com/xi/XI/System.Send.maxConsumers 10
BcAdapter.Call.maxConsumers 10
BcAdapter.Recv.maxConsumers 5
BcAdapter.Rqst.maxConsumers 10
BcAdapter.Send.maxConsumers 5
CIDXAdapter.Call.maxConsumers 10
CIDXAdapter.Recv.maxConsumers 5
CIDXAdapter.Rqst.maxConsumers 10
CIDXAdapter.Send.maxConsumers 5
CIDX_http://sap.com/xi/XI/System.Call.maxConsumers 20
CIDX_http://sap.com/xi/XI/System.Recv.maxConsumers 10
CIDX_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
CIDX_http://sap.com/xi/XI/System.Send.maxConsumers 10
EOIO.lockTimeout 5
File_http://sap.com/xi/XI/System.Call.maxConsumers 20
File_http://sap.com/xi/XI/System.Recv.maxConsumers 10
File_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
File_http://sap.com/xi/XI/System.Send.maxConsumers 10
JDBC_http://sap.com/xi/XI/System.Call.maxConsumers 20
JDBC_http://sap.com/xi/XI/System.Recv.maxConsumers 10
JDBC_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
JDBC_http://sap.com/xi/XI/System.Send.maxConsumers 10
JMS_http://sap.com/xi/XI/System.Call.maxConsumers 20
JMS_http://sap.com/xi/XI/System.Recv.maxConsumers 10
JMS_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
JMS_http://sap.com/xi/XI/System.Send.maxConsumers 10
JPR.Call.maxConsumers 10
JPR.Recv.maxConsumers 5
JPR.Rqst.maxConsumers 10
JPR.Send.maxConsumers 5
MPA.Call.maxConsumers 10
MPA.Recv.maxConsumers 5
MPA.Rqst.maxConsumers 10
MPA.Send.maxConsumers 5
Mail_http://sap.com/xi/XI/System.Call.maxConsumers 20
Mail_http://sap.com/xi/XI/System.Recv.maxConsumers 10
Mail_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
Mail_http://sap.com/xi/XI/System.Send.maxConsumers 10
Marketplace_http://sap.com/xi/XI/System.Call.maxConsumers 20
Marketplace_http://sap.com/xi/XI/System.Recv.maxConsumers 10
Marketplace_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
Marketplace_http://sap.com/xi/XI/System.Send.maxConsumers 10
RFC_http://sap.com/xi/XI/System.Call.maxConsumers 20
RFC_http://sap.com/xi/XI/System.Recv.maxConsumers 10
RFC_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
RFC_http://sap.com/xi/XI/System.Send.maxConsumers 10
RNIF11Adapter.Call.maxConsumers 20
RNIF11Adapter.Recv.maxConsumers 10
RNIF11Adapter.Rqst.maxConsumers 20
RNIF11Adapter.Send.maxConsumers 10
RNIF11_http://sap.com/xi/XI/System.Call.maxConsumers 20
RNIF11_http://sap.com/xi/XI/System.Recv.maxConsumers 10
RNIF11_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
RNIF11_http://sap.com/xi/XI/System.Send.maxConsumers 10
RNIFAdapter.Call.maxConsumers 10
RNIFAdapter.Recv.maxConsumers 5
RNIFAdapter.Rqst.maxConsumers 10
RNIFAdapter.Send.maxConsumers 5
RNIF_http://sap.com/xi/XI/System.Call.maxConsumers 20
RNIF_http://sap.com/xi/XI/System.Recv.maxConsumers 10
RNIF_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
RNIF_http://sap.com/xi/XI/System.Send.maxConsumers 10
SOAP_http://sap.com/xi/XI/System.Call.maxConsumers 20
SOAP_http://sap.com/xi/XI/System.Recv.maxConsumers 10
SOAP_http://sap.com/xi/XI/System.Rqst.maxConsumers 20
SOAP_http://sap.com/xi/XI/System.Send.maxConsumers 10
bulkCommit.threshold 100
cluster.timeout 60000
connection.0.ackListener null
connection.0.exceptionListener localejbs/AFWListener
connection.0.jms.messageSelector null
connection.0.jms.profile null
connection.0.jms.provider null
connection.0.jms.queue null
connection.0.messageListener localejbs/AFWListener
connection.0.name AFW
connection.0.pollAttempts 60
connection.0.pollInterval 60000
connection.1.ackListener localejbs/MarketPlaceAdapterListener
connection.1.exceptionListener localejbs/MarketPlaceAdapterListener
connection.1.jms.messageSelector null
connection.1.jms.profile null
connection.1.jms.provider null
connection.1.jms.queue null
connection.1.messageListener localejbs/MarketPlaceAdapterListener
connection.1.name MPA
connection.1.pollAttempts 60
connection.1.pollInterval 60000
connection.2.ackListener null
connection.2.exceptionListener localejbs/JPR
connection.2.jms.messageSelector null
connection.2.jms.profile null
connection.2.jms.provider null
connection.2.jms.queue null
connection.2.messageListener localejbs/JPR
connection.2.name JPR
connection.2.pollAttempts 60
connection.2.pollInterval 60000
connection.3.ackListener null
connection.3.exceptionListener localejbs/RNIFMessageListener
connection.3.jms.messageSelector null
connection.3.jms.profile null
connection.3.jms.provider null
connection.3.jms.queue null
connection.3.messageListener localejbs/RNIFMessageListener
connection.3.name RNIFAdapter
connection.3.pollAttempts 60
connection.3.pollInterval 60000
connection.4.ackListener null
connection.4.exceptionListener localejbs/CIDXMessageListener
connection.4.jms.messageSelector null
connection.4.jms.profile null
connection.4.jms.provider null
connection.4.jms.queue null
connection.4.messageListener localejbs/CIDXMessageListener
connection.4.name CIDXAdapter
connection.4.pollAttempts 60
connection.4.pollInterval 60000
connection.5.ackListener localejbs/BcAdapterListener
connection.5.exceptionListener localejbs/BcAdapterListener
connection.5.jms.messageSelector null
connection.5.jms.profile null
connection.5.jms.provider null
connection.5.jms.queue null
connection.5.messageListener localejbs/BcAdapterListener
connection.5.name BcAdapter
connection.5.pollAttempts 60
connection.5.pollInterval 60000
connection.6.ackListener null
connection.6.exceptionListener localejbs/TestListener
connection.6.jms.messageSelector null
connection.6.jms.profile null
connection.6.jms.provider null
connection.6.jms.queue null
connection.6.messageListener localejbs/TestListener
connection.6.name test
connection.6.pollAttempts 60
connection.6.pollInterval 60000
connection.7.ackListener null
connection.7.exceptionListener localejbs/RNIF11MessageListener
connection.7.jms.messageSelector null
connection.7.jms.profile null
connection.7.jms.provider null
connection.7.jms.queue null
connection.7.messageListener localejbs/RNIF11MessageListener
connection.7.name RNIF11Adapter
connection.7.pollAttempts 60
connection.7.pollInterval 60000
connections (name=AFW, messageListener=localejbs/AFWListener, exceptionListener=localejbs/AFWListener, pollInterval=60000, pollAttempts=60, Send.maxConsumers=5, Recv.maxConsumers=5, Call.maxConsumers=10, Rqst.maxConsumers=10)(name=MPA, messageListener=localejbs/MarketPlaceAdapterListener, exceptionListener=localejbs/MarketPlaceAdapterListener, ackListener=localejbs/MarketPlaceAdapterListener, pollInterval=60000, pollAttempts=60, Send.maxConsumers=5, Recv.maxConsumers=5, Call.maxConsumers=10, Rqst.maxConsumers=10)(name=JPR, messageListener=localejbs/JPR, exceptionListener=localejbs/JPR, pollInterval=60000, pollAttempts=60, Send.maxConsumers=5, Recv.maxConsumers=5, Call.maxConsumers=10, Rqst.maxConsumers=10)(name=RNIFAdapter, messageListener=localejbs/RNIFMessageListener, exceptionListener=localejbs/RNIFMessageListener, pollInterval=60000, pollAttempts=60, Send.maxConsumers=5, Recv.maxConsumers=5, Call.maxConsumers=10, Rqst.maxConsumers=10)(name=CIDXAdapter, messageListener=localejbs/CIDXMessageListener, exceptionListener=localejbs/CIDXMessageListener, pollInterval=60000, pollAttempts=60, Send.maxConsumers=5, Recv.maxConsumers=5, Call.maxConsumers=10, Rqst.maxConsumers=10)(name=BcAdapter, messageListener=localejbs/BcAdapterListener, exceptionListener=localejbs/BcAdapterListener, ackListener=localejbs/BcAdapterListener, pollInterval=60000, pollAttempts=60, Send.maxConsumers=5, Recv.maxConsumers=5, Call.maxConsumers=10, Rqst.maxConsumers=10)(name=test, messageListener=localejbs/TestListener, exceptionListener=localejbs/TestListener, pollInterval=60000, pollAttempts=60, Send.maxConsumers=10, Recv.maxConsumers=10, Call.maxConsumers=20, Rqst.maxConsumers=20)(name=RNIF11Adapter, messageListener=localejbs/RNIF11MessageListener, exceptionListener=localejbs/RNIF11MessageListener, pollInterval=60000, pollAttempts=60, Send.maxConsumers=10, Recv.maxConsumers=10, Call.maxConsumers=20, Rqst.maxConsumers=20)
debug HTTP
failover.reassignOnNodeLoss true
initial.delay 0
jdbc.autoConfigure false
jdbc.datasource jdbc/SAP/BC_XI_AF_MSG
jdbc.dbProductName ORACLE
jdbc.substituteEmptyString true
jdbc.supportsOpenSQL true
jdbc.supportsScrollableResultSet false
jms.0.dmq SonicMQ.deadMessage
jms.0.messageLengthKey message_length
jms.0.password ********
jms.0.provider
jms.0.url
jms.0.username
monitoring.displayDefault fromParty, toParty, status, timesFailedRetries, sentRecvTime
persistMessageRemover.checkInterval 1440
proxy.password ********
proxy.user
reassignCheck.waitTime 30
ssl.httpsHandler iaik.protocol.https.Handler
ssl.securityProvider iaik.security.provider.IAIK
ssl.serverNameCheck false
ssl.trustedCACerts.viewName TrustedCAs
startup.delay 0
syncMessageRemover.checkInterval 60
syncMessageRemover.messageCount 1000
syncMessageRemover.removeBody true
system.deafaultMaxConsumersPerQueue 10
system.failNonDelivered false
system.message.lazyRead true
system.node 23654950
system.queueConsumerThreadPoolSize 0
system.statusLog System,Send,Recv,Call,Rqst
system.transactionManager ts
test.Call.maxConsumers 20
test.Recv.maxConsumers 10
test.Rqst.maxConsumers 20
test.Send.maxConsumers 10
throughputStatistics enabled
regards
roberti -
Weblogic Message Bridge Producing Duplicate JMS Messages
We have a message bridge that we use to integrate with TIBCO. The message bridge is deployed on a WL cluster. We see duplicate JMS messages produced by the bridge when it is deployed on the cluster. We don't see this behavior when only one ManagedServer in the cluster is active,.
TIBCO version: 4.4.3 V5
WLS: 10.3.3 with patch for bug BUG8732539 (BQC6)
java -version
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Oracle JRockit(R) (build R28.0.0-679-130297-1.6.0_17-20100312-2121-linux-x86_64, compiled mode)
We are also seeing a lot of exceptions related to
<May 27, 2011 4:11:57 PM PDT> <Warning> <Connector> <BEA-190032> << eis/jms/WLSConnectionFactoryJNDIXA > ResourceAllocationException thrown by resource adapter on call to ManagedConnectionFactory.createManagedConnection(): "javax.resource.ResourceException: Failed to start the connection ">
However the bridge is active. We are monitoring the message-out count on TIBCO and we observed that we see duplicates whenever there a message is re-delivered. We opened a case with Oracle support and they suggested patch 10258751 which didn't help. The next suggestion was a patch for "Bug 11852771 - [WLS10.3.2]MISSING AND DUPLICATE IN JMS QUEUE MESSAGE MONITORING" but I don't think it will help since our JMS consumers are getting duplicate messages and this is not just a UI issue.
Here is the message behavior when we send a single message:
Timelines
11:34:17 --> Message sent to TIBCO EMS queue
11:34:17 --> WL MS3 throws an javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMFAIL : Resource manager is unavailable exception
11:34:17 --> WL MS1 says it got the message and committed the transaction
11:34:17 --> Message is visible in the JMS queue in the admin console
11:34:17 --> TIBCO EMS has still not received an ACK from WL for the message
11:34:31 --> TIBCO EMS resends the same JMS message to WL
11:34:31 --> WL MS2 says it got the message and committed the transaction
11:34:31 --> WL MS1 throws an javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMFAIL : Resource manager is unavailable exception
11:34:31 --> Message is visible in the JMS queue in the admin console (At this point there are two messages in the JMS queue)
11:34:31 --> TIBCO EMS has still not received an ACK from WL for the message
11:34:54 --> TIBCO EMS resends the same JMS message to WL
11:34:31 --> WL MS4 says it got the message and committed the transaction
11:34:31 --> WL MS2 throws an javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMFAIL : Resource manager is unavailable exception
11:34:31 --> Message is visible in the JMS queue in the admin console (At this point there are three messages in the JMS queue)
11:34:31 --> TIBCO EMS receives an ACK from WL for the message
MS04
####<May 27, 2011 11:34:54 AM PDT> <Debug> <MessagingBridgeRuntime> <> <MS04> <[ACTIVE] ExecuteThread: '109' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-6A59E8FE76C4FC4CA149> <> <1306521294611> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) received message:
JMS Message Class: TextMessage
JMSMessageID: ID:MS01.4B044DDFEE1F1:25354AD
JMSCorrelationID: null
JMSDeliveryMode: PERSISTENT
JMSDestination: Queue[SourceQueue1]
JMSExpiration: 0
JMSPriority: 4
JMSRedelivered: true
JMSReplyTo: null
JMSTimestamp: 1306521256905 (Fri May 27 11:34:16 PDT 2011)
JMSType: null
Transaction Id: BEA1-6A59E8FE76C4FC4CA149
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
####<May 27, 2011 11:34:54 AM PDT> <Debug> <MessagingBridgeRuntime> <MS04> <MS04> <[ACTIVE] ExecuteThread: '109' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-6A59E8FE76C4FC4CA149> <> <1306521294611> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) successfully sent message:
JMS Message Class: TextMessage
Old JMS MessageID: ID:MS01.4B044DDFEE1F1:25354AD
New JMS MessageID: ID:<193654.1306521294611.0>
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
MS03
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257342> <BEA-000000> <Bridge MessageBridge1 Successfully got connection to the target destination>
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257343> <BEA-000000> <Bridge MessageBridge1: both source and target adapters support XA = true>
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257343> <BEA-000000> <Bridge MessageBridge1 supportsMDBTX = false supportsXA = true async = true>
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257343> <BEA-000000> <Bridge MessageBridge1 WorkMode = Exactly-once>
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntimeVerbose> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257343> <BEA-000000> <Bridge MessageBridge1 Entering processMessages() ------ >
####<May 27, 2011 11:34:17 AM PDT> <Error> <MessagingBridge> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-3AA29116A4CA622834EA> <> <1306521257343> <BEA-200015> <An error occurred in bridge "MessageBridge1" during the transfer of messages (java.lang.Exception: javax.resource.ResourceException: Failed to setup the Resource Adapter Connection for enlistment in the transaction, Pool = 'eis/jms/WLSConnectionFactoryJNDIXA', javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMFAIL : Resource manager is unavailable
javax.transaction.xa.XAException: Internal error: XAResource 'eis/jms/WLSConnectionFactoryJNDIXA' is unavailable
at weblogic.transaction.internal.XAResourceDescriptor.checkResource(XAResourceDescriptor.java:948)
at weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:634)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1227)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:507)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:434)
at weblogic.connector.transaction.outbound.XATxConnectionHandler.enListResource(XATxConnectionHandler.java:117)
at weblogic.connector.outbound.ConnectionWrapper.invoke(ConnectionWrapper.java:218)
at $Proxy62.receive(Unknown Source)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1431)
at weblogic.jms.bridge.internal.MessagingBridge.beginForwarding(MessagingBridge.java:1006)
at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1083)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
at weblogic.connector.transaction.outbound.XATxConnectionHandler.enListResource(XATxConnectionHandler.java:200)
at weblogic.connector.outbound.ConnectionWrapper.invoke(ConnectionWrapper.java:218)
at $Proxy62.receive(Unknown Source)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1431)
at $Proxy62.receive(Unknown Source)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1431)
at weblogic.jms.bridge.internal.MessagingBridge.beginForwarding(MessagingBridge.java:1006)
at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1083)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMFAIL : Resource manager is unavailable
javax.transaction.xa.XAException: Internal error: XAResource 'eis/jms/WLSConnectionFactoryJNDIXA' is unavailable
at weblogic.transaction.internal.XAResourceDescriptor.checkResource(XAResourceDescriptor.java:948)
at weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:634)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1227)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:507)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:434)
at weblogic.connector.transaction.outbound.XATxConnectionHandler.enListResource(XATxConnectionHandler.java:117)
at weblogic.connector.outbound.ConnectionWrapper.invoke(ConnectionWrapper.java:218)
at $Proxy62.receive(Unknown Source)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1431)
at weblogic.jms.bridge.internal.MessagingBridge.beginForwarding(MessagingBridge.java:1006)
at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1083)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1185)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:507)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:434)
at weblogic.connector.transaction.outbound.XATxConnectionHandler.enListResource(XATxConnectionHandler.java:117)
... 8 more
).>
####<May 27, 2011 11:34:17 AM PDT> <Warning> <MessagingBridge> <MS03> <MS03> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521257344> <BEA-200026> <Bridge "MessageBridge1" encountered some problems in one of its adapters or underlying systems. It stopped transferring messages and will try to reconnect to the adapters shortly. (The exception caught was java.lang.Exception: MessagingBridgeException: Messaging bridge operation failed
.)>
MS02
####<May 27, 2011 11:34:31 AM PDT> <Debug> <MessagingBridgeRuntime> <MS02> <MS02> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-679AF074F381152F047C> <> <1306521271386> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) received message:
JMS Message Class: TextMessage
JMSMessageID: ID:MS01.4B044DDFEE1F1:25354AD
JMSCorrelationID: null
JMSDeliveryMode: PERSISTENT
JMSDestination: Queue[SourceQueue1]
JMSExpiration: 0
JMSPriority: 4
JMSRedelivered: true
JMSReplyTo: null
JMSTimestamp: 1306521256905 (Fri May 27 11:34:16 PDT 2011)
JMSType: null
Transaction Id: BEA1-679AF074F381152F047C
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
####<May 27, 2011 11:34:31 AM PDT> <Debug> <MessagingBridgeRuntime> <MS02> <MS02> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-679AF074F381152F047C> <> <1306521271392> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) successfully sent message:
JMS Message Class: TextMessage
Old JMS MessageID: ID:MS01.4B044DDFEE1F1:25354AD
New JMS MessageID: ID:<196075.1306521271386.0>
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
####<May 27, 2011 11:34:54 AM PDT> <Error> <MessagingBridge> <MS02> <MS02> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521294610> <BEA-200015> <An error occurred in bridge "MessageBridge1" during the transfer of messages (java.lang.Exception: weblogic.transaction.RollbackException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:495)
at weblogic.connector.transaction.outbound.XAWrapper.setTransactionTimeout(XAWrapper.java:457)
at weblogic.connector.transaction.outbound.RecoveryOnlyXAWrapper.setTransactionTimeout(RecoveryOnlyXAWrapper.java:34)
at weblogic.transaction.internal.XAResourceDescriptor.setXAResourceTransactionTimeoutIfAppropriate(XAResourceDescriptor.java:1422)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1223)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.lang.NullPointerException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:485)
... 14 more
at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:1871)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:330)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:225)
at weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManagerImpl.java:293)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1534)
at weblogic.jms.bridge.internal.MessagingBridge.beginForwarding(MessagingBridge.java:1006)
at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1083)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:495)
at weblogic.connector.transaction.outbound.XAWrapper.setTransactionTimeout(XAWrapper.java:457)
at weblogic.connector.transaction.outbound.RecoveryOnlyXAWrapper.setTransactionTimeout(RecoveryOnlyXAWrapper.java:34)
at weblogic.transaction.internal.XAResourceDescriptor.setXAResourceTransactionTimeoutIfAppropriate(XAResourceDescriptor.java:1422)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1223)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.lang.NullPointerException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:485)
... 14 more
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1185)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
... 2 more
).>
####<May 27, 2011 11:34:54 AM PDT> <Warning> <MessagingBridge> <MS02> <MS02> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521294610> <BEA-200026> <Bridge "MessageBridge1" encountered some problems in one of its adapters or underlying systems. It stopped transferring messages and will try to reconnect to the adapters shortly. (The exception caught was java.lang.Exception: MessagingBridgeException: Messaging bridge operation failed
MS01
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS01> <MS01> <[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-72BC818FE2BB8C2655C6> <> <1306521257536> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) received message:
JMS Message Class: TextMessage
JMSMessageID: ID:MS01.4B044DDFEE1F1:25354AD
JMSCorrelationID: null
JMSDeliveryMode: PERSISTENT
JMSDestination: Queue[SourceQueue1]
JMSExpiration: 0
JMSPriority: 4
JMSRedelivered: false
JMSReplyTo: null
JMSTimestamp: 1306521256905 (Fri May 27 11:34:16 PDT 2011)
JMSType: null
Transaction Id: BEA1-72BC818FE2BB8C2655C6
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
####<May 27, 2011 11:34:17 AM PDT> <Debug> <MessagingBridgeRuntime> <MS01> <MS01> <[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <BEA1-72BC818FE2BB8C2655C6> <> <1306521257542> <BEA-000000> <Bridge: MessageBridge1 (processMessages()) successfully sent message:
JMS Message Class: TextMessage
Old JMS MessageID: ID:MS01.4B044DDFEE1F1:25354AD
New JMS MessageID: ID:<195581.1306521257536.0>
<?xml version="1.0" encoding="UTF-8"?>
<commands ...
>
####<May 27, 2011 11:34:31 AM PDT> <Error> <MessagingBridge> <MS01> <MS01> <[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521271384> <BEA-200015> <An error occurred in bridge "MessageBridge1" during the transfer of messages (java.lang.Exception: weblogic.transaction.RollbackException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:495)
at weblogic.connector.transaction.outbound.XAWrapper.setTransactionTimeout(XAWrapper.java:457)
at weblogic.connector.transaction.outbound.RecoveryOnlyXAWrapper.setTransactionTimeout(RecoveryOnlyXAWrapper.java:34)
at weblogic.transaction.internal.XAResourceDescriptor.setXAResourceTransactionTimeoutIfAppropriate(XAResourceDescriptor.java:1422)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1223)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.lang.NullPointerException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:485)
... 14 more
at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:1871)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:330)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:224)
at weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManagerImpl.java:293)
at weblogic.jms.bridge.internal.MessagingBridge.processMessages(MessagingBridge.java:1534)
at weblogic.jms.bridge.internal.MessagingBridge.beginForwarding(MessagingBridge.java:1006)
at weblogic.jms.bridge.internal.MessagingBridge.run(MessagingBridge.java:1083)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: javax.transaction.SystemException: start() failed on resource 'eis/jms/WLSConnectionFactoryJNDIXA': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:495)
at weblogic.connector.transaction.outbound.XAWrapper.setTransactionTimeout(XAWrapper.java:457)
at weblogic.connector.transaction.outbound.RecoveryOnlyXAWrapper.setTransactionTimeout(RecoveryOnlyXAWrapper.java:34)
at weblogic.transaction.internal.XAResourceDescriptor.setXAResourceTransactionTimeoutIfAppropriate(XAResourceDescriptor.java:1422)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1223)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1161)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.lang.NullPointerException
at weblogic.connector.security.layer.AdapterLayer.setTransactionTimeout(AdapterLayer.java:485)
... 14 more
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1185)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:282)
at weblogic.transaction.internal.XAServerResourceInfo.enlistIfStatic(XAServerResourceInfo.java:181)
at weblogic.transaction.internal.ServerTransactionImpl.enlistStaticallyEnlistedResources(ServerTransactionImpl.java:1565)
at weblogic.transaction.internal.ServerTransactionManagerImpl.dispatchRequest(ServerTransactionManagerImpl.java:551)
at weblogic.messaging.dispatcher.DispatcherServerRef.handleRequest(DispatcherServerRef.java:132)
at weblogic.messaging.dispatcher.DispatcherServerRef.access$000(DispatcherServerRef.java:34)
at weblogic.messaging.dispatcher.DispatcherServerRef$2.run(DispatcherServerRef.java:111)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
... 1 more
).>
####<May 27, 2011 11:34:31 AM PDT> <Warning> <MessagingBridge> <MS01> <MS01> <[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306521271384> <BEA-200026> <Bridge "MessageBridge1" encountered some problems in one of its adapters or underlying systems. It stopped transferring messages and will try to reconnect to the adapters shortly. (The exception caught was java.lang.Exception: MessagingBridgeException: Messaging bridge operation failed
.)>@Tom, thanks for your hints and sorry for my delayed response. Here are my answers to your queries…
-- Does messaging work, without any exceptions, when only one bridge is active?
It works fine when only one node of the cluster is active. -- I'm not familiar with Tibco monitoring, but I suspect that the "message-out count on TIBCO" stat doesn't always imply that there's a duplicate message. It could mean that an initial message delivery attempt failed, it's transaction rolled back, and the message was redelivered. Redelivery after a failure is normal - the original delivered message, if one actually made it onto a WebLogic queue, should be discarded by WebLogic when its failed transaction eventually rolls back (before its ever handed to a consumer application).
You are absolutely right. The mismatch in count was due to redelivery attempts. We could see that as soon as we turned on trace in TIBCO-- The posted stack traces look like they may be JVM stdout/stderr scrapes, which are often automatically truncated, for example, see the "... 8 more" listed at the bottom of one of the stack traces below. There might be more information in the full stack trace for the "Resource manager is unavailable" exception. Full traces are logged in each WebLogic Server's log files.
I tried redirecting the output and even set "stdout Stack Trace Depth:" to -1 but I am not albe to see the complete trace.-- The setTransactionTimeout NullPointerException error message looks like it is collateral damage from the original reported "Resource manager is unavailable" exception that's thrown earlier during the enlistResource() step. If you are truly getting duplicates, then I think this is a clue that the bridge is somehow (stupidly) trying to proceed with forwarding the message despite the fact that there's no current transaction on the thread...
I have a case open with Oracle support and we have extensive debugging turned on. Here is what the engineer came back with. There is an issue with XA transactions in JCA adapter when it talks to third party providers like TIBCO. Since the resource names are not unique across the cluster, a XA transaction is started on multiple nodes in the cluster for the same message. All transactions abort but the message is persisted and TIBCO redelivers the message since the connection is severed when the transaction aborts. We were able to reproduce this scenario at will with our setup and Oracle was able to reproduce the issue too. -- Can you check the state of the message on the WebLogic Q? The fact that a message is shown on the WL console doesn't necessarily mean that the state of the message is visible (it could be waiting for the transaction to commit - something that apparently never happens). On the console: Services -> Messaging -> JMS Modules -> [module name] -> [destination name].
Our consumer was getting duplicate messages. This confirms that the console was not giving us incorrect information. The console showed that 2/3/4 messages were in "current" state when we expected only one message. -- Make sure that the bridge's Tibco destination is a Tibco queue not a Tibco topic (it's happened).
Yeah we made sure it was a Queue. -- Can you post your bridge and destination stanzas from your config.xml?
Will try to do this shortly but have to strip out lot of information. -- I vaguely recall that there was a problem with untyped connection factories in Tibco at one point (not sure). You might want to try changing the Tibco configuration to spin-up "XA Queue Connection Factory" instead of just an "XA Connection Factory" (plus modify the bridge configuration accordingly).
We tried this but it didn't eliminate this issue. One thing we did notice however was that the number of rollbacks were greatly reduced.
Here is the workaround that Oracle is suggesting: Create pinned destinations and assign it to a distributed queue. This will keep the transactions local and hence eliminate the duplicate transactions and rollback. We tried it and it eliminated the dups issue. However I discovered another bug during my testing. Our cluster messaging mode is "Unicast" and message forwarding from zero consumer queues to queues with consumers fails. We have four/five nodes in the cluster and two consumer threads. I observed that the zero consumer queues always forward to one queue in the cluster that has consumers. The forwarding stalls when I bring down the node that has consumers. Again, we were able to reproduce this issue at will and Oracle has acknowledged that it is a bug. They've reported that Unicast fails without any warning whereas Multicast fails with a warning in the log. We have observed this message forwarding stalling even with UDQs.
I have another case open for an issue where our message bridges that transfer messages from TIBCO to WLS stalls after sometime. The status is shown as active but it is not doing anything. The server instance shows stuck threads trying to roll back a message. TIBCO logs indicate that it is receiving a request to rollback a transaction but it is not finding the transaction to rollback. The only way to recover from this condition is to restart the cluster. I am fairly confident that he root case for this issue is again the XA transaction issue. I changed our UDQ to pinned destination in one environment and we are no longer seeing dups or bridge stalling there. Hopefully Oracle can resolve the message forwarding issue.Thanks again…. -
JMS message from one application to the another appl in the same container?
I have a question.
Can I send JMS message from one application to the another application in the same container?
If yes ...can you provide me info.
I want to know in regards of EJB2.0 and EJB3.0.
Thanks,
RahulSo the receiver application is a Message Driven Bean, right?
In your container, do you have a mean to ensure that it is correctly configured to listen the desired destination? Like an administrative console that would show the destinations, the number of pending messages, the number of active consummers...?
btw what is the destination type? A queue or a topic? -
Hello All,
I have two WLS domains. One WLS 10.3.X, another is WLS 8.1.X
Each domain have one queue each (e.g. 10.3.X => Q1 & 8.1.X => Q2) and the 10.3.X domain have been configured with a MessageBridge to deliver the messages from Q1 to Q2
Each domain have ConnectionFactories XA enabled and the Bridge adapter is also set to "eis.jms.WLSConnectionFactoryJNDIXA"
Now the Bridge is showing active, but if we put a test message in Q1, it is taking long to be delivered to Q2. There is no error in the logs (Debug Flgs have also been enabled).
Sometimes it is also not being delivered at all.
During this, the status of the message in the Q1 is showing "receive transaction".
So can anyone tell me :
1> what are the valid status of any message in wls jms queues and what are the meaning of them.
2> what is the lifecycle of a message.
Thanks in advancehttp://docs.oracle.com/cd/E24329_01/apirefs.1211/e24391/weblogic/jms/extensions/JMSMessageInfo.html#JMSMessageInfo(long,%20int,%20java.lang.String,%20long,%20java.lang.String,%20weblogic.jms.extensions.WLMessage,%20java.lang.String,%20boolean)
"visible - The message is visible on the destination and may be consumed.
send - The message is pending because it is associated with an in progress send operation.
receive - The message is pending because it has been received by a consumer but not yet acknowledged.
transaction - The message is associated with a transactional “send” or “receive” operation that is in progress.
ordered - The message belongs to a Unit-of-Order (UOO).
delayed - The message is pending because it has a scheduled delivery time or is otherwise delayed.
expired - The message has expired and is no longer available for consumption.
redelivery-count-exceeded - The message has exceeded the redelivery count set for the destination and is no longer available for consumption.
paused - The message is pending because of a destination pause operation .
sequenced - The message that is part of a SAF sequence.
unit-of-work-component - The message is part of an Unit-of-Work (UOW).
deleted - The message is marked for deletion."
You can enable message life cycle logging
- http://docs.oracle.com/cd/E24329_01/web.1211/e24385/troubleshoot.htm#i1128815
"The message life cycle is an external view of the events that a JMS message traverses through once it has been accepted by the JMS server, either through the JMS APIs or the JMS Message Management APIs. Message life cycle logging provides an administrator with easy access to information about the existence and status of JMS messages from the JMS server viewpoint. In particular, each message log contains information about basic life cycle events such as message production, consumption, and removal."
There is also a WLST script provided here (Re: Example using of getMessage(s) with JMS error queues? for browsing messages on a given destination. -
Development Provisioning Profiles Stuck on PENDING
Hi Guys,
I have just recently added an iPad3 to a provisioning profile.
Can anyone explain why my certificates are stuck on pending? Is this one for apple support?
Thanks.I've spent all day on this and haven't progressed an iota. I am in provisioning profile h*ll.
It's not like I'm a rank beginner - my friend and I have 8 apps up at the app store already and have been doing this since September. But since these certs expired, I just haven't been able to get moving again.
Everything works just fine in the simulator, but I can't generate a build for the iPhone or the app store without the error mentioned in my first message occurring.
Nobody else is having problems after their certs or developer provisioning profiles expire when they try to generate and install new ones?
I downloaded the most recent iPhone Developer Program User Guide v2.4 and went page-by-page, step-by-step doing everything from scratch, but it just doesn't seem to work anymore.
...sigh...
doug -
JMS messages not being redelivered correctly ..
Hi Folks
I am having this weird problem with JMS using workshop/weblogic 8.1 SP2. I am
using a JMS control from within my pageflow to post a JMS message to a predefined
JMS queue (created by workshop for every web project . Mine is called testweb.queue.AsyncDispatcher
as my web project is called testweb). Since pageflows cannot receive asysnchronus
callbacks , I have not implemented the callback handler for teh JMS control as
I just need to be able to post from my Pageflow . So far so good.
I have an MDB listening to testweb.queue.AsyncDispatcher and that basically tries
to open a URL connection to a thirdparty web site and if connect is not successful
it should keep trying after every 10 seconds. I have the MDB throw an EJBException
(to simulate message redelivery) . The problem is that the message is redelivered
exactly 8 times and then I get the below message on the console :-
<May 24, 2004 10:18:00 AM EDT> <Error> <WLW> <F1234PiyushK> <cgServer> <ExecuteThread:
'14' for queue: 'weblogic.kernel.Default'> <<anonymous>> <BEA1-001D75CA6E6EB041A149>
<000000> <onMessageException: java.lang.Exception: Unexpected message format>
I also have an MDB listening to the error queue testweb.queue.AsyncDispatcher_error
but its onMessage is never executed .
I am stuck on this for the last efw days so appreciate any help. Should I post
a message to the JMS queue directly from a session bean insetad of from a page
flow ?
my JMS queue config parameters are mentioned below :-
Time To Deliver Delivery 5000
Delivery Mode Override - No Override
Expiration Policy - Redirect
Redelivery Delay Override - 5000
Redelivery Delay Override - 10
Error Destination - testweb.queue.AsyncDispatcher_error
The other parameter values are the default values and I have not changed themI am having the same problem with a simple WLI timer channel that posts a string message called myMessage every 20 seconds (what I defined in the WLI console).
This works like a charm.
In my process.jpd file, the timer subscription is followed by a JMS control, with a sendTextMessage(myMessage) which sends the message to a queue
@jc:jms
send-type="queue" send-jndi-name="JmsTimerProcessWeb.queue.AsyncDispatcher"
receive-type="queue" receive-jndi-name="NightlySiteProcess.queue.AsyncDispatcher"
connection-factory-jndi-name="weblogic.jws.jms.QueueConnectionFactory"
All goes well while sending but the messages tell otherwise:
=== Timer sent message = This is the WLI message
=== JMS sent message = This is the WLI message
<Aug 24, 2005 1:19:30 PM MDT> <Error> <WLW> <000000> <onMessageException: java.lang.Exception: Unexpected message format>
What does this mean?
I tried to add a project with a Message-bean, I cannot get the string message. I tried with an ObjectMessage, the same scenario everytime.
Any suggestions? -
Need suggestion on implementing JMS message error recovery
Hi,
Our application has a JMS topic where we publish application events. Now, there can be scenarios where the consumers cannot process the message due to some infrastructure issues and would error out. We need a way so that those messages can be reprocessed again later. we are thinking of the following design for JMS message error recovery
1. Use a persistent TOPIC (this would ensure guaranteed delivery)
2. Configure a error destination on JMS topic e.g a jms queue
3. Have an error handling MDB listening to the Error destination. An error handling MDB would dequeue the errored messages from error destination and persist it to a Data base "error" table..
4. Provide a mechanism to republish those messages to topic (e.g a scheduler or admin ui or a command line utility) .. The messages would deleted from database "error" table and published to topic again....
A. Are there any issues with the above design which we need to handle?
B. Are there any additional steps required in a Cluster environment with a distributed topic and distribute error destination? (our error mdb will have one-copy-per-application setting)
B. From a performance angle, Is it OK to use persistent TOPIC ? Or will it better to persist the message to the db table and then publish it as a non persistent message ... ? (But i guess the performance should be more or less the same in both of these approaches)
C. Are there any other recommended design patterns for error recovery of JMS messages
Please advise.
Regards,
ArifThanks Tom !
We may not be able to go with the approach of delaying/pausing redelivery of the messsage because
1. Pausing entire MDB approach: Our MDB application consumes messages generated by different producers and our MDB needs to continue processing the messages even if messages corresponding to one producer is erroring out
2. Redelivery delay : This would only delay the retry of an errored message. But there would still be a problem if the message fails during all retries (i.e redelivery limit count). We don't want to lose this message. In our case, It is possible that a particular message cannot be processed due to unavailability of a third party system for hours or may be a day.
Basically, i am looking on approaches for a robust and performant error recovery/retry framework for our application (refer details in my first post on this thread) while fully making use of all features provided by middleware (WLS). Please advise.
Regards,
Arif -
Lost (and found) JMS - messages
Using WLS 6.1 and have experienced that JMS messages has not been delivered to
its consumer correctly. We are interested in any suggestions to what has happend,
what can be done to avoid the problem in the future, or perhaps how we can find
out more about the problem.
We have a message driven bean A which sends messages to the queue Q, of which
the message driven bean B is the only consumer.
What we recently have experienced is that some of the messages delivered by A
to Q never are delivered to B. But the messages seems to be persisted correctly,
because after a restart they reappear!
Following is a more detailed description of the scenario:
1. A message is sendt with DeliveryMode=PERSISTENT and TimeToLive=0. QueueSender.send
is invoked and returns normally.
2. The consumers never gets the message (B.onMessage i never invoked with this
message).
3. Point 1-2 is repeated for 13 different messages over a period when ~100 similar
messages are delivered normally.
4. The server is then restarted.
5. In the starup log it is indicated that 44 records are found in the JMSStore.
6. When the server restarts, the "lost" messages gets delivered to B.
After the server restart we have not experienced the problem again.
Logs does not show anything unusual about the previous restart.
Other details:
There is always exatly 1 instance of A running, and 5-20 instances of B. A implements
and uses a pool of javax.jms.XAQueueSession instances, which shares a single instance
of javax.jms.XAQueueConnection. The aknowledge-mode for B is "Auto-aknowledge".
We're using Weblogic Server 6.1.3.0 / Red Hat Linux 7.3.
Any help on this problem is appreciated!
I believe we've seen just the behaviour you describe on 6.1 SP2, but only on a
multi-cpu box (Solaris), and only under load. We can't reproduce the problem reliably,
and get no error messages/warnings anywhere, so very little to send to customer
support; by the time we notice this, the problem's long passed, so a thread dump's
of no use.
We're going to try and see how SP5 (which has loads of JMS fixes) pans out.
But if you find anything out, could you post here?
simon.
"Harald Stendal" <[email protected]> wrote:
>
>Using WLS 6.1 and have experienced that JMS messages has not been delivered
>to
>its consumer correctly. We are interested in any suggestions to what
>has happend,
>what can be done to avoid the problem in the future, or perhaps how we
>can find
>out more about the problem.
>
>We have a message driven bean A which sends messages to the queue Q,
>of which
>the message driven bean B is the only consumer.
>What we recently have experienced is that some of the messages delivered
>by A
>to Q never are delivered to B. But the messages seems to be persisted
>correctly,
>because after a restart they reappear!
>
>Following is a more detailed description of the scenario:
>
>1. A message is sendt with DeliveryMode=PERSISTENT and TimeToLive=0.
>QueueSender.send
>is invoked and returns normally.
>2. The consumers never gets the message (B.onMessage i never invoked
>with this
>message).
>3. Point 1-2 is repeated for 13 different messages over a period when
>~100 similar
>messages are delivered normally.
>4. The server is then restarted.
>5. In the starup log it is indicated that 44 records are found in the
>JMSStore.
>6. When the server restarts, the "lost" messages gets delivered to B.
>
>After the server restart we have not experienced the problem again.
>
>Logs does not show anything unusual about the previous restart.
>
>Other details:
>There is always exatly 1 instance of A running, and 5-20 instances of
>B. A implements
>and uses a pool of javax.jms.XAQueueSession instances, which shares a
>single instance
>of javax.jms.XAQueueConnection. The aknowledge-mode for B is "Auto-aknowledge".
>
>We're using Weblogic Server 6.1.3.0 / Red Hat Linux 7.3.
>
>Any help on this problem is appreciated!
>
Maybe you are looking for
-
When updating itunes, it keeps saying invalid drive E, why does it look for drive e?
When updating itunes, it keeps saying invalid drive E, why does it look for drive e?
-
Default Calendar Option being ignored in iOS7
I seem to be having an issue with the Calendar, after upgrading to iOS7. (7.0.2) I have multiple calendars synced to my Exchange email account. In previous versions (iOS6 for example), I have been able to change the default Calendar that all new appo
-
Hi experts, Could anybody explain to me what is BO Live Office? I need to set up my crystal reports and webi and my admin asks me to download live office. on sap site, i found a link to free down load of live office, but in the list there is none. i
-
I'm trying to send a text file through Airport to another Mac mini. I think Personal File Sharing would allow me to put files I want to transfer in my Public folder, then retrieve them on the other Mac mini, but when I tried to do this, I had some pr
-
Load content of folder located in /content/dam/
Hi, I'm trying to load the entire content of folder "/content/dam/myFolder". Is it possible? I've tried this piece of Java code File folder = new File("/content/dam/myFolder/"); File[] listOfFiles = folder.listFiles(); if(listOfFiles == null){%>