Sun Java Web Server 6.1SP4 deadlock, stops responding

Question

Hi, i have a java web app on a Java Web SErver 6.1SP4 and when there are a lot of users the web server hangs, it stops responing. After a while (from 2 to 15 minutes) it start responding again. This serves neither dinamic nor static content. CPU usage is 100%. Just stoping and restarting the web server instance it works properly (even with the same or greater number of clients).
Reading many posts some one wrote to use pstack to see whats happening. I did it, but i can't interpret results (i think there is a deadlock but don't know if thats true and how to resolve it).
Any hint will be appreciated
There are a lot of blocks like this:
----------------- lwp# 3468 / thread# 3468 --------------------
fe11f950 lwp_mutex_lock (918c18)
fd692384 __1cNObjectMonitorGenter26MpnGThread__v_ (5000, 525c, 5000, 50dc, 4800, 4af0) + 2d8
fd6324d4 __1cSObjectSynchronizerKfast_enter6FnGHandle_pnJBasicLock_pnGThread__v_ (e104f604, e104f77c, 831ff50, 0, 35d654, fd6328ec) + 68
fd632954 __1cQinstanceRefKlassZacquire_pending_list_lock6FpnJBasicLock__v_ (e104f77c, fd970000, e7330000, 4491d4, fd61bc2c, 0) + 78
fd63167c __1cPVM_GC_OperationNdoit_prologue6M_i_ (e104f764, 4400, fd970000, 2cdc0, 4a6268, 1) + 38
fd62e0b0 __1cIVMThreadHexecute6FpnMVM_Operation__v_ (e104f764, 80e9b0, 0, 0, 1, 0) + 90
fd52c2a4 __1cbCTwoGenerationCollectorPolicyRmem_allocate_work6MIii_pnIHeapWord__ (2e368, fd9c29ec, fd9c297c, fd931a26, 4800, 4998) + 160
fd522940 __1cNinstanceKlassRallocate_instance6MpnGThread__pnPinstanceOopDesc__ (f3c09020, 831ff50, 81e33e0, 4000, 4178, 0) + 180
fd7b7744 __1cQjava_lang_StringQbasic_create_oop6FpnQtypeArrayOopDesc_ipnGThread__pnHoopDesc__ (f0a5e910, 0, 831ff50, 0, 1, 1) + a8
fd63228c jni_NewStringUTF (831ffe4, 605b9c8, fd587824, 80e9f8, 4800, 4998) + b8
fdafdefc __1cKNSJavaUtilQsetStringElement6FpnHJNIEnv__pnN_jobjectArray_Ipkc_v_ (831ffe4, ace7e4, 2, 605b9c8, 4, 605b828) + 34
fdb068dc __1cONSAPIConnectorOgetRequestInfo6MpnHJNIEnv__pnN_jobjectArray_4pnK_jintArray__v_ (e104fabc, 831ffe4, ace7e4, ace7e0, ace7dc, 4) + 9c
fdb042e4 __1cONSAPIConnectorHservice6MpnRJ2EEVirtualServer__i_ (e104fabc, ea9560, 8, 84b9464, 6d137e0, ace7dc) + 260
fdb02734 service_j2ee (3c1c0, 84b97f0, 84b9868, 0, fdb11d57, 132414) + 40
ff1cf994 __1cNfunc_exec_str6FpnKFuncStruct_pnGpblock_pnHSession_pnHRequest__i_ (668, 3c1c0, 84b97f0, 84b9868, 0, 0) + 248
ff1d0db4 INTobject_execute (77308, 84b97f0, 84b9868, 0, 37ff8, 80d290) + 5e8
ff1d5de4 INTservact_service (84b97f0, 84b9868, ff2e7b58, 0, 0, ff2e7b30) + 4d8
ff1d64f4 INTservact_handle_processed (84b97f0, 84b9868, 20, 2, 605bee0, 76858) + 158
ff218a9c __1cLHttpRequestUUnacceleratedRespond6Mpc_v_ (84b9750, ff2e7b7c, 2f48, 1bb, 84b9868, 84b97f0) + 3c8
ff21818c __1cLHttpRequestNHandleRequest6MpnGnetbuf__i_ (84b9750, 6059290, 605b3f0, 605b3d8, 2000, 60592f0) + 62c
ff216588 __1cNDaemonSessionDrun6M_v_ (84b9348, 2000, ff2ed7bc, 0, 0, ff2ed774) + 17c
ff106dec ThreadMain (84b9348, 84fcef0, 3, 0, 1, 4b8) + 24
fedd0028 ptroot (84fcef0, 0, 0, 0, 20000, fede8d70) + d0
fe165c94 lwpstart (0, 0, 0, 0, 0, 0)

Answer 1

hi
sorry for the delayed response.
well, the problem that you are trying to run into is very common when your server is getting good amount of load and uses large amount of servlet/jsp web applications running.
here is a url that you might find it a useful read
http://java.sun.com/docs/hotspot/gc1.4.2/index.html#4.%20Types%20of%20Collectors|outline
if you do not want to take this trouble reading this document and want a quick solution, you do can do couple of things:
- first understand how to size the heap
http://java.sun.com/docs/hotspot/gc1.4.2/index.html#3.%20Sizing%20the%20Generations|outline
- understand that the system is paused because jvm is busy doing garbage collection before allowing the web server to continue.
- this can be avoided by properly tuning the jvm to how you think the server should behave. unfortunately, there is no miracle one one line to fix this issue
- by careful and decent analysis of your system, you can tune your jvm properly that it can scale very well without any glitches.
though, you will need to take the pain to learn the garbage collection acronyms and see what's best suited for you.
- first, you want to know what is happening inside the jvm before you proceed to optimize . you can do this by adding this option
-verbose:gc -XX:+PrintGCDetails
How to add this option in the server.xml , is by doing something like
start admin server -> manager your server -> click Java tab
-> click on JVM Option in the left side
-> Add this JVM option -verbose:gc -XX:+PrintGCDetails
-> restart the server
-> note, you do NOT want to leave the above option in the production environment as this will tremendously slow your application and this is meant to be only for development purposes.
- if your server is a multi processor server (has more than 1 CPU) and you want less pause time, you can typically try adding this jvm option to your server and restarting it.
-XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
here are few other short cut , jvm options that you also try in your devleopment environment before going to production.
Server application(Like Web Server 6.1) running alone on a large multi-processor server with 4gb of physical memory.
#java -server -XX:+AggressiveHeap
Two applications running on a large multi-processor server with 4gb of physical memory. Each java application instance is allocated a part of total system memory by an explicit specification of the maximum and minimum heap sizes.
#java -server -XX:+AggressiveHeap -Xms1024m -Xmx1024m
(again, do your research before you use this option in the production environment)