Multiple EJB clients blocking

Question

I tried to ask this question a few months ago, but I must not have given enough
details
for anyone to even guess what my problem might be. So I created a simple example
to
demonstrate what's happening.
I have a cluster of 2 WL servers (7.0) running on Linux. When I call an EJB multiple
times at once from different threads in a client application, sometimes all the
threads
finish in a few milliseconds, but other times at least one of the threads will
pause
for 60 seconds before finishing. There are no error or warning indicators in any
server
log files. I'm trying to do something very simple: add one to a number in a database
table without letting any two clients get the same number.
I'm using DB2 as my database and COM.ibm.db2.jdbc.app.Driver as my driver. I have
a
JDBC connection pool and a DataSource, which I'm using in a stateless session
bean.
I lock the table in exclusive mode, query the current number, and add one to it.
I'm
using container-managed transactions, so I'm relying on the container to commit
the
update when the method ends. (I tried using a bean-managed transaction, and I
got the
same results.)
For a long time I believed the problem must be with DB2, but when I put the exact
same
code in a JSP, it works just fine. So I know that DB2 can do what I'm asking it
to do.
I also tried it on a Windows NT server, and it behaves exactly the same. Comparing
thread dumps of the two servers in the cluster taken during one of these 60 second
wait periods doesn't show any obvious problems: the two servers both have exactly
the
same threads going, and they all seem pretty reasonable to me.
Here is the code for my EJB:
package com.davisvision.cv;
import java.sql.*;
import javax.ejb.*;
public class SimpleBean implements SessionBean {
SessionContext ctx;
public void ejbCreate() {
public void ejbPostCreate() {
public void ejbRemove() {
public void ejbActivate() {
public void ejbPassivate() {
public void setSessionContext(SessionContext c) {
ctx = c;
public int nextNumber() {
String tableName = "TEST.PROVIDER_SPAWN";
Connection con = null;
int num = 0;
try {
con = CvDatabase.getConnection();
con.setAutoCommit(false);
Statement stmt = con.createStatement();
stmt.executeUpdate("LOCK TABLE " + tableName + " IN EXCLUSIVE MODE");
ResultSet rs = stmt.executeQuery("SELECT NEXT_OFFICE_NUM FROM "
+ tableName + " WHERE PROVIDER_TYPE = 'DAVIS2'");
if (rs.next()) {
num = rs.getInt(1);
num++;
int updated = stmt.executeUpdate("UPDATE " + tableName
+ " SET NEXT_OFFICE_NUM = " + num
+ " WHERE PROVIDER_TYPE = 'DAVIS2'");
catch(Exception e) {
num = -1;
Logger.writeDated(Logger.FATAL, "SimpleBean.nextNumber: " + e);
finally {
CvDatabase.close(con);
return num;
And here is my client application:
package test;
import com.davisvision.cv.*;
import java.util.*;
import javax.ejb.*;
import javax.naming.*;
import javax.rmi.PortableRemoteObject;
public class TestSimpleBean implements Runnable {
static InitialContext ctx;
public int taskNum;
static void getContext() {
Properties env = new Properties();
env.put(Context.INITIAL_CONTEXT_FACTORY, "weblogic.jndi.WLInitialContextFactory");
env.put(Context.PROVIDER_URL, "t3://weblogictest:7055");
try {
ctx = new InitialContext(env);
catch(NamingException e) {
System.out.println("Unable to obtain an initial JNDI context:
" + e);
public void run() {
System.out.println("Thread " + taskNum + " started.");
try {
Object obj = ctx.lookup("ejb-cv-Simple");
SimpleHome h = (SimpleHome) PortableRemoteObject.narrow(obj, SimpleHome.class);
SimpleRemote r = h.create();
int n = r.nextNumber();
System.out.println("Thread " + taskNum + " next number = " + n);
catch(Exception e) {
System.out.println("Error in thread " + taskNum + ": " + e);
System.out.println("Thread " + taskNum + " ended.");
public static void main(String[] args) {
getContext();
int n = 3;
TestSimpleBean x = null;
Thread t = null;
// start each thread
for (int i = 0; i < n; i++) {
x = new TestSimpleBean();
x.taskNum = i;
t = new Thread(x);
t.start();
System.out.println("" + n + " threads started.");
My deployment descriptors are as simple as I could make them:
<?xml version="1.0"?>
<!DOCTYPE ejb-jar PUBLIC
"-//Sun Microsystems, Inc.//DTD Enterprise JavaBeans 2.0//EN"
"http://java.sun.com/dtd/ejb-jar_2_0.dtd">
<ejb-jar>
<enterprise-beans>
<session>
<ejb-name>Simple</ejb-name>
<home>com.davisvision.cv.SimpleHome</home>
<remote>com.davisvision.cv.SimpleRemote</remote>
<ejb-class>com.davisvision.cv.SimpleBean</ejb-class>
<session-type>Stateless</session-type>
<transaction-type>Container</transaction-type>
</session>
</enterprise-beans>
<assembly-descriptor>
<container-transaction>
<method>
<ejb-name>Simple</ejb-name>
<method-name>
</method-name>
</method>
<trans-attribute>
Required
</trans-attribute>
</container-transaction>
</assembly-descriptor>
</ejb-jar>
<?xml version="1.0"?>
<!DOCTYPE weblogic-ejb-jar PUBLIC
"-//BEA Systems, Inc.//DTD WebLogic 6.0.0 EJB//EN"
"http://www.bea.com/servers/wls600/dtd/weblogic-ejb-jar.dtd" >
<weblogic-ejb-jar>
<weblogic-enterprise-bean>
<ejb-name>Simple</ejb-name>
<jndi-name>ejb-cv-Simple</jndi-name>
</weblogic-enterprise-bean>
</weblogic-ejb-jar>
This is from part of a production application, so to make sure our users aren't
getting
random 60-second pauses, I have the production bean deployed on just one server
in the
cluster with just one bean in the pool. But I really need to find the solution
to this
problem before development goes much further. Any guesses, sugestions, or conjectures
would be most welcome.
- gary

Answer 1

Oops, I didn't notice that I only attached one file last time. The thread dump
from the other server in the cluster is attached.
My (off-topic) Linux problem must be something else: Signaling the parent
java process was one of the first things I tried. It may have to do with the
way that we start WL (in the background with nohup).
- gary
Rob Woollen <[email protected]> wrote:
Gary Bradshaw wrote:
Thanks for your response, Rob. I've attached the thread dumps, hopingthat they
will be easier to read that way. They are from a Windows NT system:Thanks. Are you sure this is the right dump? The server looks
completely idle in this dump.
I haven't
yet found a way to get a thread dump on my Linux system. (I know Ishould say
"kill -3 <processId>", but each WLS has at least 72 process IDs. MaybeI just
haven't found the right one yet.)Try ps -ef --forest  .  It'll show you graphically with a little ascii
art the parent process/thread.
-- Rob
Your suggestion to do the update first would work for this particularexample
since I know I always want to add 1 to the number, but in other similarsituations
I will want to vary the update based on the values I find in the row.I'm looking
for a clean way to safely do multi-user updates with JDBC. (For larger,more important
tables I'm using entity beans).
- gary
Rob Woollen <[email protected]> wrote:
Can you post the thread dumps?
Also usually people do something like this:
update mytable set count = count + 1
select count from mytable
-- Rob
Gary Bradshaw wrote:
I tried to ask this question a few months ago, but I must not havegiven enough
details
for anyone to even guess what my problem might be. So I created a
simple
example
to
demonstrate what's happening.
I have a cluster of 2 WL servers (7.0) running on Linux. When I callan EJB multiple
times at once from different threads in a client application, sometimesall the
threads
finish in a few milliseconds, but other times at least one of the
threads
will
pause
for 60 seconds before finishing. There are no error or warning indicatorsin any
server
log files. I'm trying to do something very simple: add one to a numberin a database
table without letting any two clients get the same number.
I'm using DB2 as my database and COM.ibm.db2.jdbc.app.Driver as mydriver. I have
a
JDBC connection pool and a DataSource, which I'm using in a statelesssession
bean.
I lock the table in exclusive mode, query the current number, and
add
one to it.
I'm
using container-managed transactions, so I'm relying on the containerto commit
the
update when the method ends. (I tried using a bean-managed transaction,and I
got the
same results.)
For a long time I believed the problem must be with DB2, but when
I
put the exact
same
code in a JSP, it works just fine. So I know that DB2 can do what
I'm
asking it
to do.
I also tried it on a Windows NT server, and it behaves exactly thesame. Comparing
thread dumps of the two servers in the cluster taken during one ofthese 60 second
wait periods doesn't show any obvious problems: the two servers bothhave exactly
the
same threads going, and they all seem pretty reasonable to me.
Here is the code for my EJB:
package com.davisvision.cv;
import java.sql.*;
import javax.ejb.*;
public class SimpleBean implements SessionBean {
SessionContext ctx;
public void ejbCreate() {
public void ejbPostCreate() {
public void ejbRemove() {
public void ejbActivate() {
public void ejbPassivate() {
public void setSessionContext(SessionContext c) {
ctx = c;
public int nextNumber() {
String tableName = "TEST.PROVIDER_SPAWN";
Connection con = null;
int num = 0;
try {
con = CvDatabase.getConnection();
con.setAutoCommit(false);
Statement stmt = con.createStatement();
stmt.executeUpdate("LOCK TABLE " + tableName + " IN EXCLUSIVEMODE");
ResultSet rs = stmt.executeQuery("SELECT NEXT_OFFICE_NUM
FROM "
+ tableName + " WHERE PROVIDER_TYPE = 'DAVIS2'");
if (rs.next()) {
num = rs.getInt(1);
num++;
int updated = stmt.executeUpdate("UPDATE " + tableName
+ " SET NEXT_OFFICE_NUM = " + num
+ " WHERE PROVIDER_TYPE = 'DAVIS2'");
catch(Exception e) {
num = -1;
Logger.writeDated(Logger.FATAL, "SimpleBean.nextNumber:" + e);
finally {
CvDatabase.close(con);
return num;
And here is my client application:
package test;
import com.davisvision.cv.*;
import java.util.*;
import javax.ejb.*;
import javax.naming.*;
import javax.rmi.PortableRemoteObject;
public class TestSimpleBean implements Runnable {
static InitialContext ctx;
public int taskNum;
static void getContext() {
Properties env = new Properties();
env.put(Context.INITIAL_CONTEXT_FACTORY, "weblogic.jndi.WLInitialContextFactory");
env.put(Context.PROVIDER_URL, "t3://weblogictest:7055");
try {
ctx = new InitialContext(env);
catch(NamingException e) {
System.out.println("Unable to obtain an initial JNDIcontext:
" + e);
public void run() {
System.out.println("Thread " + taskNum + " started.");
try {
Object obj = ctx.lookup("ejb-cv-Simple");
SimpleHome h = (SimpleHome) PortableRemoteObject.narrow(obj,SimpleHome.class);
SimpleRemote r = h.create();
int n = r.nextNumber();
System.out.println("Thread " + taskNum + " next number= " + n);
catch(Exception e) {
System.out.println("Error in thread " + taskNum +
" + e);
System.out.println("Thread " + taskNum + " ended.");
public static void main(String[] args) {
getContext();
int n = 3;
TestSimpleBean x = null;
Thread t = null;
// start each thread
for (int i = 0; i < n; i++) {
x = new TestSimpleBean();
x.taskNum = i;
t = new Thread(x);
t.start();
System.out.println("" + n + " threads started.");
My deployment descriptors are as simple as I could make them:
<?xml version="1.0"?>
<!DOCTYPE ejb-jar PUBLIC
"-//Sun Microsystems, Inc.//DTD Enterprise JavaBeans 2.0//EN"
"http://java.sun.com/dtd/ejb-jar_2_0.dtd">
<ejb-jar>
<enterprise-beans>
<session>
<ejb-name>Simple</ejb-name>
<home>com.davisvision.cv.SimpleHome</home>
<remote>com.davisvision.cv.SimpleRemote</remote>
<ejb-class>com.davisvision.cv.SimpleBean</ejb-class>
<session-type>Stateless</session-type>
<transaction-type>Container</transaction-type>
</session>
</enterprise-beans>
<assembly-descriptor>
<container-transaction>
<method>
<ejb-name>Simple</ejb-name>
<method-name>
</method-name>
</method>
<trans-attribute>
Required
</trans-attribute>
</container-transaction>
</assembly-descriptor>
</ejb-jar>
<?xml version="1.0"?>
<!DOCTYPE weblogic-ejb-jar PUBLIC
"-//BEA Systems, Inc.//DTD WebLogic 6.0.0 EJB//EN"
"http://www.bea.com/servers/wls600/dtd/weblogic-ejb-jar.dtd" >
<weblogic-ejb-jar>
<weblogic-enterprise-bean>
<ejb-name>Simple</ejb-name>
<jndi-name>ejb-cv-Simple</jndi-name>
</weblogic-enterprise-bean>
</weblogic-ejb-jar>
This is from part of a production application, so to make sure ourusers aren't
getting
random 60-second pauses, I have the production bean deployed on justone server
in the
cluster with just one bean in the pool. But I really need to find
the
solution
to this
problem before development goes much further. Any guesses, sugestions,or conjectures
would be most welcome.
- gary------------------------------------------------------------------------
Full thread dump:
"Thread-7" daemon prio=5 tid=0x868400 nid=0x1f8 waiting on monitor[0x136ff000..0x136ffdc0]
     at java.lang.Thread.sleep(Native Method)
     at com.davisvision.cv.CvProperties.run(CvProperties.java:153)
     at java.lang.Thread.run(Thread.java:479)
"ListenThread.Default" prio=5 tid=0x8647d0 nid=0x1f0 runnable [0x136bf000..0x136bfdc0]
     at java.net.PlainSocketImpl.socketAccept(Native Method)
     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
     at java.net.ServerSocket.implAccept(ServerSocket.java:238)
     at java.net.ServerSocket.accept(ServerSocket.java:217)
     at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
     at weblogic.t3.srvr.ListenThread.run(ListenThread.java:263)
"ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=5 tid=0x84d220nid=0x1e9 waiting on monitor [0x1312f000..0x1312fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=5tid=0x84c040 nid=0x1e8 waiting on monitor [0x130ef000..0x130efdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=5tid=0x84c580 nid=0x1e7 waiting on monitor [0x130af000..0x130afdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=5tid=0x84cec0 nid=0x1a1 waiting on monitor [0x1306f000..0x1306fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=5tid=0x84b370 nid=0x1a0 waiting on monitor [0x1302f000..0x1302fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"CoreHealthMonitor" daemon prio=5 tid=0x829e30 nid=0x1e6 waiting onmonitor [0x12fef000..0x12fefdc0]
     at java.lang.Thread.sleep(Native Method)
     at weblogic.t3.srvr.CoreHealthMonitorThread.run(CoreHealthMonitorThread.java:114)
"ExecuteThread: '14' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x820420nid=0x1e5 waiting on monitor [0x12faf000..0x12fafdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '13' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x81f2d0nid=0x1e4 waiting on monitor [0x12f6f000..0x12f6fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '12' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x81fec0nid=0x1e3 waiting on monitor [0x12f2f000..0x12f2fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '11' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x81eac0nid=0x1e2 waiting on monitor [0x12eef000..0x12eefdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '10' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x81d6c0nid=0x1e1 waiting on monitor [0x12eaf000..0x12eafdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '9' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x81c2d0nid=0x1e0 waiting on monitor [0x12e6f000..0x12e6fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '8' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x81cec0nid=0x1df waiting on monitor [0x12e2f000..0x12e2fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '7' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x81bac0nid=0x1de waiting on monitor [0x12def000..0x12defdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '6' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x81a6c0nid=0x1dd waiting on monitor [0x12daf000..0x12dafdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '5' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x8192d0nid=0x1dc waiting on monitor [0x12d6f000..0x12d6fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '4' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x819ec0nid=0x1db waiting on monitor [0x12d2f000..0x12d2fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '3' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x818990nid=0x1da waiting on monitor [0x12cef000..0x12cefdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '2' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x817630nid=0x1d9 waiting on monitor [0x12caf000..0x12cafdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x816150nid=0x1d8 waiting on monitor [0x12c6f000..0x12c6fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'JmsDispatcher'" daemon prio=5 tid=0x816690nid=0x1d7 waiting on monitor [0x12c2f000..0x12c2fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '2' for queue: 'weblogic.transaction.AsyncQueue'" daemonprio=5 tid=0x811cc0 nid=0x1d6 waiting on monitor [0x12bef000..0x12befdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: 'weblogic.transaction.AsyncQueue'" daemonprio=5 tid=0x8104c0 nid=0x1d5 waiting on monitor [0x12baf000..0x12bafdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'weblogic.transaction.AsyncQueue'" daemonprio=5 tid=0x80f020 nid=0x1d4 waiting on monitor [0x12b6f000..0x12b6fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"Thread-4" prio=5 tid=0x80e440 nid=0x1d3 waiting on monitor [0x12b2f000..0x12b2fdc0]
     at java.lang.Object.wait(Native Method)
     at java.util.TimerThread.mainLoop(Timer.java:427)
     at java.util.TimerThread.run(Timer.java:380)
"ExecuteThread: '1' for queue: 'DRS'" daemon prio=5 tid=0x80d700 nid=0x1d2waiting on monitor [0x12aef000..0x12aefdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'DRS'" daemon prio=5 tid=0x80b160 nid=0x1d1waiting on monitor [0x12aaf000..0x12aafdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'Multicast'" daemon prio=5 tid=0x80a8b0nid=0x1d0 waiting on monitor [0x12a6f000..0x12a6fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: 'Replication'" daemon prio=5 tid=0x8072f0nid=0x1cf waiting on monitor [0x12a2f000..0x12a2fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: 'Replication'" daemon prio=5 tid=0x807430nid=0x1c9 waiting on monitor [0x129ef000..0x129efdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"LDAPConnThread 10.1.7.26:7001" daemon prio=5 tid=0x7f56c0 nid=0x1cdrunnable [0x1296f000..0x1296fdc0]
     at java.net.SocketInputStream.socketRead(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:85)
     at java.io.BufferedInputStream.fill(BufferedInputStream.java:181)
     at java.io.BufferedInputStream.read(BufferedInputStream.java:199)
     at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
     at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:420)
"LDAPConnThread 10.1.7.26:7001" daemon prio=5 tid=0x7f35c0 nid=0x1cbrunnable [0x128ef000..0x128efdc0]
     at java.net.SocketInputStream.socketRead(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:85)
     at java.io.BufferedInputStream.fill(BufferedInputStream.java:181)
     at java.io.BufferedInputStream.read(BufferedInputStream.java:199)
     at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
     at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:420)
"LDAPConnThread 10.1.7.26:7001" daemon prio=5 tid=0x7edc70 nid=0x1carunnable [0x128af000..0x128afdc0]
     at java.net.SocketInputStream.socketRead(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:85)
     at java.io.BufferedInputStream.fill(BufferedInputStream.java:181)
     at java.io.BufferedInputStream.read(BufferedInputStream.java:199)
     at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
     at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:420)
"VDE Transaction Processor Thread" prio=2 tid=0x7e21c0 nid=0x1c5 waitingon monitor [0x1282f000..0x1282fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:306)
     at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:192)
"ExecuteThread: '1' for queue: '__weblogic_admin_rmi_queue'" daemonprio=5 tid=0x7a7020 nid=0x1c4 waiting on monitor [0x127df000..0x127dfdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: '__weblogic_admin_rmi_queue'" daemonprio=5 tid=0x7d7750 nid=0x1c3 waiting on monitor [0x1279f000..0x1279fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '1' for queue: '__weblogic_admin_html_queue'" daemonprio=5 tid=0x7c26b0 nid=0x1c2 waiting on monitor [0x1273f000..0x1273fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: '__weblogic_admin_html_queue'" daemonprio=5 tid=0x7c2bf0 nid=0x1c1 waiting on monitor [0x126ff000..0x126ffdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"TimeEventGenerator" daemon prio=5 tid=0x7c00d0 nid=0x1c0 waiting onmonitor [0x126bf000..0x126bfdc0]
     at java.lang.Object.wait(Native Method)
     at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
     at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:139)
     at java.lang.Thread.run(Thread.java:479)
"ExecuteThread: '1' for queue: '_weblogic_dgc_queue'" daemon prio=5tid=0x7bfac0 nid=0x1bf waiting on monitor [0x1267f000..0x1267fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '0' for queue: '_weblogic_dgc_queue'" daemon prio=5tid=0x7be5c0 nid=0x1be waiting on monitor [0x1263f000..0x1263fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"HighPriority TimeEventGenerator" daemon prio=9 tid=0x7bee10 nid=0x1bdwaiting on monitor [0x125ff000..0x125ffdc0]
     at java.lang.Object.wait(Native Method)
     at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
     at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:139)
     at java.lang.Thread.run(Thread.java:479)
"SpinnerRandomSource" daemon prio=5 tid=0x7bc270 nid=0x1bc waitingon monitor [0x125bf000..0x125bfdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.security.SpinnerRandomBitsSource.run(SpinnerRandomBitsSource.java:60)
     at java.lang.Thread.run(Thread.java:479)
"ExecuteThread: '14' for queue: 'default'" daemon prio=5 tid=0x7b9050nid=0x1bb runnable [0x1257f000..0x1257fdc0]
     at weblogic.socket.NTSocketMuxer.getNextSocket(Native Method)
     at weblogic.socket.NTSocketMuxer.processSockets(NTSocketMuxer.java:534)
     at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:23)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:213)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:189)
"ExecuteThread: '13' for queue: 'default'" daemon prio=5 tid=0x7b8050nid=0x1ba runnable [0x1253f000..0x1253fdc0]
     at weblogic.socket.NTSocketMuxer.getNextSocket(Native Method)
     at weblogic.socket.NTSocketMuxer.processSockets(NTSocketMuxer.java:534)
     at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:23)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:213)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:189)
"ExecuteThread: '12' for queue: 'default'" daemon prio=5 tid=0x7b8840nid=0x1b9 runnable [0x124ff000..0x124ffdc0]
     at java.net.PlainDatagramSocketImpl.receive(Native Method)
     at java.net.DatagramSocket.receive(DatagramSocket.java:387)
     at weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:158)
     at weblogic.cluster.MulticastManager.execute(MulticastManager.java:336)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:213)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:189)
"ExecuteThread: '11' for queue: 'default'" daemon prio=5 tid=0x7b7360nid=0x1b8 waiting on monitor [0x124bf000..0x124bfdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '10' for queue: 'default'" daemon prio=5 tid=0x7b7ec0nid=0x1b7 waiting on monitor [0x1247f000..0x1247fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '9' for queue: 'default'" daemon prio=5 tid=0x7b6ac0nid=0x1b6 waiting on monitor [0x1243f000..0x1243fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '8' for queue: 'default'" daemon prio=5 tid=0x7b56c0nid=0x1b5 waiting on monitor [0x123ff000..0x123ffdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '7' for queue: 'default'" daemon prio=5 tid=0x7b4360nid=0x1b4 waiting on monitor [0x123bf000..0x123bfdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:146)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
"ExecuteThread: '6' for queue: 'default'" daemon prio=5 tid=0x7b4ec0nid=0x1b3 waiting on monitor [0x1237f000..0x1237fdc0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:415)
     at weblogic.kernel.ExecuteThread.