WLS 8.1sp3 blocked threads halt MDB processing

Under Weblogic Server 8.1 SP3 JMS consumers stop processing the JMS queues. This only started happening following our upgrade from SP2. We had numerous test baselines as well as months of production data to validate this.
          Specifically it appears that internal weblogic threads are blocked, preventing the MDBs (JMS consumers) from being dispatched. This doesn't immediately impact the user experience, but eventually unprocessed JMS messages accumulate and consume heap. You can figure out the rest.
          We had 5 MDBs working on this particular JMS queue, so under moderate traffic all 5 would need to be blocked before the problem was realized.
          As a work around, you'll need to request a patch from BEA. CR188391 appears to resolve this issue. You must request this however. BEA will not contact you - regardless of functional or economic impact.
          Thanks
          -Sean

Sean,
I've forwarded your comments internally. Usually, one can look in the release notes of future service-packs to see what's been fixed. I would think upgrading from SP3 to sp4, or even sp5 (the latest) would also have fixed the problem.
Tom, BEA

Similar Messages

How to interrupt MDB processing

Hi
We have a problem with web service call from MDB. Due to some network problems the call can hang for 30 minutes and any no way of setting timeout on JAX-WS port nor sun.net works. Whats more there is a silent transaction timeout on JTA (with no exception) and when web service call returnes after such a long time with success, the message is rolled back to JMS queue.
I wonder if there is any wat to set timeout on MDB processing that would casue exception and let us control this behaviour. When there is more messages like described above it causes our queue processing to stall and gives us real headache !
please help

Thank you a lot for replies
You both are right.
As we went down to implementation details it appeared that there is no way (at lest not brutal way) to stop MDB processing. I believe that fiddling with threads in appserver environment is asking for troubles, so I decided to find the source of first problem: no timeout to WS processing.
To our suprise it was caused by... migration from NetBeans 6.5 to 6.7. Somehow build scripts were affected and some jars were added to ejb jar, which were not there while using NB 6.5. Some of these jars were JAX-WS jars.
It seams that timout does not work when using JAX-WS shipped with NB 6.7 but there is no problem using libraries shipped with Glassfish 2.1.
When we removed JAX-WS from our deployment we got controls of timeouts back in our hands.
thank you again

Fetching records in a block(db) for mass processing

My user interface inputs commit frequency for intermittent commits. We reference this to # of records processed & then issue commit. [whilst processing against millions of records]
Now, we want to process all records belonging to set of data blocks to avoid contention issues (gc/ other locking issues) & then issue commit.
So the question is, how do I query records belonging to a particular block OR identify those blocks that hold my data.
Any pointer is greatly appreciated.
thanks
Kiran.

kkulkarn wrote:
This is needed for a DW instance wherein we select recordset from a OLTP instance running 4 node RAC. The user then inputs a condition based on which the records are selected.
We are commiting based on parameters defined by DBA's - the commit frequency. This increases contention especially if the records are stored in contiguous location. Add to this the RAC, the GC traffic is huge.
So what we want to do is to grab all the records in a particular block & read the whole block & then process to DW. I was looking at a query that can get me the block# involved in this process.
Could you clarify:
Is it the OLTP that is on RAC, or the DW, or both ?
Which type of contention - can you show us the critical wait information - and is this on the OLTP or RAC system, and is it the OLTP or RAC system where the GC traffic is huge ?
You seem to be asking how you can specify a query in an OLTP system that is going to visit many table blocks - but "pre-sort" the data in some way so that it visits each table block only once, and visits the table blocks in order so that you don't revisit the same table block and run the risk of extra interconnect traffic on the OLTP system. Is this a correct interpretation of your requirement ?
Does your transfer process do something like:
counter := 0;
for r in (select from OLTP system) loop
    insert into DW system values (r.thing....);
    counter := counter + 1;
    if counter = {limit} then
        commit;
        counter := 0;
    end if;
end loop;
commit;Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan

Maximum number of threads in a process

Hello,
we are currently running on Solaris 9 an application compiled
with C++ compiler SunStudio 8.
Our application crash after running 2990 concurrent threads with the system
error
ENOMEM (Not enough space).
In the same time the real memory free was around 8 Gb.
So what kind of resource was missing ?
Is there any limit of concurrent threads inside a process ?
Thanks for any tips.
Yaakov Berkovitch
[email protected]

Hello,
we are currently running on Solaris 9 anapplication
compiled
with C++ compiler SunStudio 8.
Our application crash after running 2990concurrent
threads with the system
error
ENOMEM (Not enough space).
In the same time the real memory free was around 8
Gb.
So what kind of resource was missing ?
Is there any limit of concurrent threads inside a
process ?It could the memory available to the process. You
don't say if it's a 32bit or 64bit application. If
it's 32 bit, it could well be out of memory and/or
stack.
A+
PaulOur application is 32-bit.
But regarding the memory the server has a big mount of free memory so I cannot understand why the OS failed creating a new thread.
BTW, the system call that failed was thr_create.
A+ says that "vous parlez francais !!!!"

Invoice BLOCK & HOLD through EDI Processing

Hi SAP Gurus,
Can you please let me know as to how to BLOCK & HOLD Invoices while processing through EDI.
Regards,
Yasir

Hi ,
You can use the program RBDMANI2 which will reprocess all the IDOCs (EDIinvoices )that have failed . You can schedule this program in the background and it will post all failed invoices which have the GR posted later .
We had the same scenario and we have scheduled this program as a daily job at night .
Regards,
Hari.

Stopping a blocked Thread

I need to stop a Thread, or interrupt a blocked I/O.
I'm making a call to a library that I do not own and cannot change. The library makes a HTTP connection to a service and waits for a response. As far as I know, the library uses BufferedReader to read the service response. If the service is down, then no response is ever returned, and the BufferedReader read simply blocks.
I cannot change the library to be intelligent, of course, and don't have very good luck getting the service support to fix their stuff. But I have to use the library. How can I interrupt the blocked I/O read?
What I've tried is as follows, but I'm not convinced it's the right way. My application is a long-running application that does not exit very often, so I'm concerned also about gc not properly recycling the resources due to still having the blocked thread still there (and this has impacted subsequent executions, even though I use local variables where I can).
        // Create a new Thread to wait on the submission
        Thread t = new Thread() {
            public void run() {
                response = event.submit();
        t.setDaemon(true);
        t.start();
        try {
            t.join(timeOut);
            msg = "Attempt to connect to ESM Server timed out";
        catch (InterruptedException e) {
            msg = e.getMessage();
        }The other option I was thinking of, was coding my call to ESM server in a separate Class with a main, and invoking it via Runtime.exec. That way, at least I'm 100% sure that the code exits, and I can use System.out/err to receive the "output".
Given that Thread.stop() is deprecated, and that you theoretically cannot interrupt blocked I/O, how have others solved this problem?
Thanks,
Matt

...you theoretically cannot interrupt blocked I/O, how have others solved this problem?I've done it your way. Each thread does an access. If it takes too long, then it may never respond and I ignore it. I also log the error condition and try to notify and administrator.
Threads probably take up about a megabyte of memory so I may run out of memory it the problem doesn't get fixed by manual intervention.
Your idea of using Runtime means that you create a new JVM for each ESM. If a thread takes up a megabyte, the new JVM takes up many, many megabytes. Instead of running out of memory in the JVM, you may run out of memory in the Box.
Every situation is different. My thread approach works for my application. What works best for you is your call.

Obtaining monitor lock of a blocked thread

Is there a way to obtain the monitor lock (reference to the object) of a blocked thread? Something like Object Thread.getMonitorLockedObject()Also when you get the locked object it would be useful if you can get reference to the thread which is holding it, e.g.
Thread Object.getLockingThread() although the latter could be achieved through iterating over all threads and calling boolean Thread.holdsLock(Object)Such methods would help very much in debugging deadlocks and other concurrency problems.

<pre>
Is there a way to obtain the monitor lock (reference
to the object) of a blocked thread? Something like
Object Thread.getMonitorLockedObject()
Not that I'm aware of, and a quick glance through the APi didn't look promising.
Also when you get the locked object it would be
useful if you can get reference to the thread which
is holding it, e.g.
Thread Object.getLockingThread()although the latter could be achieved through
iterating over all threads and calling boolean
Thread.holdsLock(Object)
Again, no. And your work-around wouldn't work, since the holdsLock method is static and can only be called for the current thread.
Such methods would help very much in debugging
deadlocks and other concurrency problems.There already are some useful methods for that in the Thread class, e.g. getState() and getAllStackTraces().
</pre>

How to see and clear the blocked threads

Where can I see how many threads are currently running and how many threads are blocked and how many are in idle status.
Where can I see all these information and if I want to kill the blocked threads how to do that
Thanks in advance

This can be found in Weblogic Log folder..

After downloading a movie to my iPad it is blocked with the word "processing" under the movie icon. What should I do?

After downloading a movie to my iPad it is blocked with the word "processing" under the movie icon. What should I do?

Do you not want a backup of the iTunes content on your computer in the event the iPad is lost, stolen, or simply fails?

Cfcache template cf10 issue. When I use it thousands ( 15000) of threads remains in process coldfusion.exe until server crash.

Hi,
Using cfcache to cache templates for a short period of time, the number of threads in coldfusion.exe process increase unexpectedly. Moreover, these threads remains in process (view resource monitor). We don't know why. Maybe an xml config file of CF10 take control of these threads?
When I use this code:
<cflock name="prodDetail#URL.ID#" type="exclusive" timeout="10000">
<cfcache timespan="#createTimeSpan(0,0,0,30)#" key="prodDetail#code#" id="prodDetail#code#" name="prodDetail#code#">
</cfcache>
</cflock>
Thanks in advance.

Hi,
This is Windows audit feature, I suppose that you have known that, actually, when you configure this audit policy, audit event is generated when sensitive privilege requests are made. This includes following objects:
• A privileged service is called.
• One of the following privileges are called:
◦Act as part of the operating system
◦Back up files and directories
◦Create a token object
◦Debug programs
◦Enable computer and user accounts to be trusted for delegation
◦Generate security audits
◦Impersonate a client after authentication
◦Load and unload device drivers
◦Manage auditing and security log
◦Modify firmware environment values
◦Replace a process-level token
◦Restore files and directories
◦Take ownership of files or other objects
So the culprit is not Windows system, this should be caused by a massive privileged operations. As we can see, the privileged operation is
SeTakeOwnershipPrivilege. In my environment, if I run some opearations such as backup, there are
records as expected, you suspected this is related to Acronis services, if these services require so many privileged operations, Windows would audit this via its mechanism.
If you have any feedback on our support, please click
here
Alex Zhao
TechNet Community Support

Blocked threads

Hi, I need help to solve the following problem:
I have a thread T calling a method x() of a class C.
The problem is that, by my point of view, the method C.x() is an atomic function and it could be blocking or it could take too much time.
Now I need to kill the thread after a while (i.e. after timeout expiration).
I know that the Thread.close() method is deprecated and that, in this case, Thread.interrupt() can't solve the problem (I can't use status variables).
I've also read something about java.nio package but I don't know how to use it.
Is there anybody can help me?
Thanks

I'm implementing a simple server that gets request from clients.
Clients open a socket connection on a well known port.
When the server receives a new connection request it creates a new thread.
This thread is in charge to receive, parse the commands coming from the client (via socket). Based on the request it calls the related method for the class. Normally the method is a call for a CORBA object's function. And the problem is that the response (that is, the function has finished) could come after a while and it is out of my control (for example CORBA objects has problems or the request itself takes time).
After timeout expiration I need to kill all the blocked threads, because the server must handle a maximum number of threads, and if I don't kill blocked threads no other clients (requests) are admitted.
thanks for your help
d.

Does the JRE 1.5 is supported for the WLS 8.1SP3 client

Hi,
Does the JRE 1.5 is supported for the WLS 8.1SP3 client
Regards,
Ahamad

Hi,
Does the JRE 1.5 is supported for the WLS 8.1SP3 client
Regards,
Ahamad

CPU going 100%. Can blocked thread may be a reason?

Hi Team,
May be a dumb question. In my application around 40 threads are in blocked state and CPU is going about 100%. Is blocked threads may be reason for that? As threads are blocked, CPU should not be busy entertaining them, that is the reason why this question came in mind. Can somebody clear my this doubt.
Thanks in advance.

I encounter the same problem, although I run Berkeley DB (Ver 6.0.20, C#) under .NET Framework and Windows server 2008(x64). Any BDB application of win32 runs well but will encounter trouble under platform of x64 when compile BDB to x64, even though the DLL compiled and linked with win32. The bug is that Berkeley DB take amount of memory as the size of databases and regardless of cacheSize. My estimation is that all memory for BDB malloced and NOT freed.

I need block diagrams for audio processing effects such as echo,flanger,vibrato,chorus for implementing in labview?

i need block diagrams for audio processing effects such as echo,flanger,vibrato,chorus for implementing in labview? pls i need it very quickly........

Hi komalamani,
i need it very quickly...
So you better start coding soon!
Best regards,
GerdW
CLAD, using 2009SP1 + LV2011SP1 + LV2014SP1 on WinXP+Win7+cRIO
Kudos are welcome

Change request #063743 & WLS 6.1sp3

          Change request #063743 in the WLS sp4 resolved JMS issue list says:
          "Fixed a problem with Message Driven Beans not acknowledging object messages"
          Are there any more details on this? Would this affect any MDB onMessage() operation,
          or just those that throw exceptions? Would this affect MapMessages as well as
          ObjectMessages? Also, is this related to the negative numbers I see in the "Bytes
          Pending" column after my message is processed? What sort of problems do the unacknowledged
          messages usually cause for people?
          Any thoughts would be much appreciated!
          -JT


Actually I think the text on this one is wrong, or at least
          misleading. I just read the internal notes on the problem
          and it seems that the actual problem was that
          topic MDBs were not acknowledging messages if max-beans-in-free-pool
          was explicitly set to 1 in the weblogic descriptor, regardless
          of message type. Which leads me to guess you could work-around
          the problem temporarily by making the topic MDB transactional or
          increasing max-beans-in-free-pool higher than 1. Not 100%
          sure. Contact customer support to nail it down. (The newsgroups
          are not run by customer support - instead they are run by various
          volunteers.)
          Note that SP4 introduces another problem that breaks
          acknowledgement rather than fixes it. You will do best to
          go directly to SP5 (do not pass go, do not collect $200). Or
          contact customer support for the patch on SP4. I do not
          have the change request number for that one.
          Tom
          Jason Taylor wrote:
          > Change request #063743 in the WLS sp4 resolved JMS issue list says:
          > "Fixed a problem with Message Driven Beans not acknowledging object messages"
          >
          > Are there any more details on this? Would this affect any MDB onMessage() operation,
          > or just those that throw exceptions? Would this affect MapMessages as well as
          > ObjectMessages? Also, is this related to the negative numbers I see in the "Bytes
          > Pending" column after my message is processed? What sort of problems do the unacknowledged
          > messages usually cause for people?
          Messages never get deleted. The JMS server eventually runs out
          of memory or or starts to throw quota exceptions.
          When the JMS server is rebooted, messages re-appear.
          >
          > Any thoughts would be much appreciated!
          >
          > -JT

WLS 8.1sp3 blocked threads halt MDB processing

Similar Messages

Maybe you are looking for