WLS dies during stress testing

Question

We're using JMeter to send continuous requests to the server. It creates
1500 threads and each requests Hello.jsp (see below) 500 times.
The server runs fine for about 6.5 minutes (serviced ~120000 requests)
before locking up. We let it sit for a while and about 4 minutes later, it
starts spewing out the following (bottom part of weblogic.log)
Fri May 19 10:48:37 GMT-04:00 2000:<I> <WebLogicServer> WebLogic Server
started
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp: init
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp: param
verbose initialized to: true
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp: param
packagePrefix initialized to: jsp_servlet
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp: param
compileCommand initialized to: /opt/Solaris_JDK_1.2.2_05a/bin/javac
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp: param
srcCompiler initialized to weblogic.jspc
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp: param
superclass initialized to null
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp: param
workingDir initialized to: /opt/ejbserver/weblogic/myserver/classfiles
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp: param
pageCheckSeconds initialized to: 1
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp:
initialization complete
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> Generated
java file:
/opt/ejbserver/weblogic/myserver/classfiles/jsp_servlet/testpages/hello.java
Fri May 19 10:51:10 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=15123,localport=7111]''
Fri May 19 10:51:10 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=15172,localport=7111]''
Fri May 19 10:51:10 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=11312,localport=7111]''
Fri May 19 10:51:20 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=14368,localport=7111]''
Fri May 19 10:51:20 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=14385,localport=7111]''
Fri May 19 10:51:24 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=15294,localport=7111]''
Fri May 19 10:51:27 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=15291,localport=7111]''
Fri May 19 10:51:55 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=28068,localport=7111]''
Fri May 19 10:51:56 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=28385,localport=7111]''
Fri May 19 10:52:00 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=28421,localport=7111]''
Fri May 19 10:53:45 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=63711,localport=7111]''
Fri May 19 10:53:45 GMT-04:00 2000:<W> <ListenThread> Connection rejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=63709,localport=7111]''
Fri May 19 10:58:26 GMT-04:00 2000:<E> <WebLogicServer> Exception invoking
weblogic.time.server.ScheduledTrigger@d2573f
java.lang.RuntimeException: <UTIL> UnsyncCircularQueue was full!
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Compiled Code)
at java.lang.Throwable.<init>(Compiled Code)
at java.lang.Exception.<init>(Compiled Code)
at java.lang.RuntimeException.<init>(Compiled Code)
at weblogic.utils.UnsyncCircularQueue.expandQueue(Compiled Code)
at weblogic.utils.UnsyncCircularQueue.put(Compiled Code)
at weblogic.kernel.ExecuteThreadManager.execute(Compiled Code)
at weblogic.kernel.Kernel.execute(Compiled Code)
at weblogic.time.common.internal.ScheduledTrigger.private_execute(Compiled
Code)
at weblogic.time.server.ScheduledTrigger.private_execute(Compiled Code)
at weblogic.time.common.internal.TimeTable.execute(Compiled Code)
at weblogic.time.common.internal.TimeEventGenerator.run(Compiled Code)
at java.lang.Thread.run(Thread.java:479)
We get a steady stream of this exact same RuntimeException for about 20
minutes before the whole thing dies.
Any ideas would be great!
Jason
The setup is the following:
Sun E250 with 1 CPU at 400MHz, 512 MB
Solaris 2.6
WLS 5.1 with SP3
Solaris Performance Pack is used
Sun JDK 1.2.2
min heap size is 128MB and max as 128MB with options
"-native -verbosegc"
other changed to the OS:
rlim_fd_max=4096
rlim_fd_cur=4096
tcp_close_wait_interval=10000
tcp_flush_fin_wait_2=10000
No paging/swapping takes place during the test.
Hello.jsp is as follows:
<html>
<head>
<title>A simple JSP file</title>
</head>
<body>
<%
out.print("<p><b>Hello World!</b>");
%>
</body>
</html>

Answer 1

Comments inline.
Cheers - Wei
Jason <[email protected]> wrote in message
news:[email protected]...
Thanks, that definitely sheds some light on the problem.
We monitored the server and came up with the following. The queue isindeed
growing and probably accounts for the eventual failure (does anyone know
what the limit is?). The throughput is constant but just can't keep upwith
the requests. Another thing is that the heap slowly increases along with
the queue and it took a while for it to hit 100% and have GC occur.I am not sure about the limit. Either 32K or 64K, I guess. If you monitor
your queue lenght frequently, you might have a chance to spot that limit.
Every request need memory to hold. So the heap slowly increases along with
the queue.
>
Now, we duplicated the test but accessed a Servlet directly instead of a
JSP. This servlet simply did some out.print(...) statements to displaythe
same page as Hello.jsp. This ran without a hitch. We had about 5 times
more throughput and the queue wasn't growing out of control. Also, theheap
would reach 100% much quicker and GC occured more frequently. I can't see
how handling a JSP request is that much more resource intensive than a
Servlet request...
We then tried the original test again (accessing JSP) but this timeremoved
Service pack 3. We were running straight WLS 5.1 (no service packs).
Everything behaved nicely (just like the Servlet test did). Could it then
be that SP3 introduced a bug relating to how JSP requests are handled?Well, if this was the case, you might need to address this with support.
>
Finally, all the tests ran without changing the default number of execute
threads (15). I don't think increasing the number will help seeing how
short-lived these requests are. I think more execute threads would just
increase the context-switching.I agree. I've seen 15 to 60 threads in production. Might be worth a try to
use larger number of threads in your test to see if it alleviated the
problem.
>
Thanks again,My pleasure.
Jason
Wei Guan <[email protected]> wrote in message
news:[email protected]...
WebLogic has its interal request queue to enqueue all requests. Therequest
refers to an unit of work for weblogic to process, such as a triggeredtime
service, http request, etc. The finite number of worker threads defined
in
your weblogic.properties file will dequeue the requests inside therequest
queue and process the work.
There is an upper limit on the size of this queue. The message you gotmeans
that your test hit that limit. In practice, if there were lots of
requests
inside that queue, every request need to wait for lots of time beforebeing
processed (the queue is FIFO I believed). Open your WebLogic Console and
monitor the size of the queue, in your test or in your production, if
you
see the size of the queue increases constantly, there is somethingwrong.
You might need to increase the number of threads to handle the load. If
tuning up the number of threads doesn't help, you might need useweblogic
clustering (or add more servers in your clustering) to share the load.
WebLogic engineers might be able to lift up the upper limit of the sizeof
the queue. However, the larger upper limit on the queue might not helpyou
in practice. Tuning your configuration and making sure the number of
requests inside the queue do not increase constantly might be a good
practice to follow.
My 2 cents.
Cheers - Wei
Jason <[email protected]> wrote in message
news:[email protected]...
We're using JMeter to send continuous requests to the server. It
creates
1500 threads and each requests Hello.jsp (see below) 500 times.
The server runs fine for about 6.5 minutes (serviced ~120000 requests)
before locking up. We let it sit for a while and about 4 minutes
later,
it
starts spewing out the following (bottom part of weblogic.log)
Fri May 19 10:48:37 GMT-04:00 2000:<I> <WebLogicServer> WebLogic
Server
started
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp:init
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp:param
verbose initialized to: true
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp:param
packagePrefix initialized to: jsp_servlet
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp:param
compileCommand initialized to: /opt/Solaris_JDK_1.2.2_05a/bin/javac
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp:param
srcCompiler initialized to weblogic.jspc
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp:param
superclass initialized to null
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp:param
workingDir initialized to: /opt/ejbserver/weblogic/myserver/classfiles
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp:param
pageCheckSeconds initialized to: 1
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General> *.jsp:
initialization complete
Fri May 19 10:48:47 GMT-04:00 2000:<I> <ServletContext-General>
Generated
java file:
/opt/ejbserver/weblogic/myserver/classfiles/jsp_servlet/testpages/hello.java
Fri May 19 10:51:10 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=15123,localport=7111]''
Fri May 19 10:51:10 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=15172,localport=7111]''
Fri May 19 10:51:10 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=11312,localport=7111]''
Fri May 19 10:51:20 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=14368,localport=7111]''
Fri May 19 10:51:20 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=14385,localport=7111]''
Fri May 19 10:51:24 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=15294,localport=7111]''
Fri May 19 10:51:27 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=15291,localport=7111]''
Fri May 19 10:51:55 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=28068,localport=7111]''
Fri May 19 10:51:56 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=28385,localport=7111]''
Fri May 19 10:52:00 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=28421,localport=7111]''
Fri May 19 10:53:45 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=63711,localport=7111]''
Fri May 19 10:53:45 GMT-04:00 2000:<W> <ListenThread> Connectionrejected:
'Login timed out after: '5000' ms on socket:
'Socket[addr=47.187.230.161/47.187.230.161,port=63709,localport=7111]''
Fri May 19 10:58:26 GMT-04:00 2000:<E> <WebLogicServer> Exceptioninvoking
weblogic.time.server.ScheduledTrigger@d2573f
java.lang.RuntimeException: <UTIL> UnsyncCircularQueue was full!
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Compiled Code)
at java.lang.Throwable.<init>(Compiled Code)
at java.lang.Exception.<init>(Compiled Code)
at java.lang.RuntimeException.<init>(Compiled Code)
at weblogic.utils.UnsyncCircularQueue.expandQueue(Compiled Code)
at weblogic.utils.UnsyncCircularQueue.put(Compiled Code)
at weblogic.kernel.ExecuteThreadManager.execute(Compiled Code)
at weblogic.kernel.Kernel.execute(Compiled Code)
atweblogic.time.common.internal.ScheduledTrigger.private_execute(Compiled
Code)
at weblogic.time.server.ScheduledTrigger.private_execute(Compiled
Code)
at weblogic.time.common.internal.TimeTable.execute(Compiled Code)
at weblogic.time.common.internal.TimeEventGenerator.run(CompiledCode)
at java.lang.Thread.run(Thread.java:479)
We get a steady stream of this exact same RuntimeException for about20
minutes before the whole thing dies.
Any ideas would be great!
Jason
The setup is the following:
Sun E250 with 1 CPU at 400MHz, 512 MB
Solaris 2.6
WLS 5.1 with SP3
Solaris Performance Pack is used
Sun JDK 1.2.2
min heap size is 128MB and max as 128MB with options
"-native -verbosegc"
other changed to the OS:
rlim_fd_max=4096
rlim_fd_cur=4096
tcp_close_wait_interval=10000
tcp_flush_fin_wait_2=10000
No paging/swapping takes place during the test.
Hello.jsp is as follows:
<html>
<head>
<title>A simple JSP file</title>
</head>
<body>
<%
out.print("<p><b>Hello World!</b>");
%>
</body>
</html>