Weblogic - 1g- Application servers are going down with lot of struck threds

Question

Hi All,
We are seeing issue with one of the non prod server, as soon as we start the server we see lot struck threads in logs and server console is opening veryslowly after some time application server is going down. Please see the few logs in TD.
[2012-07-10 06:14:08,063] [ICOG802944407]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <SAXOutputter> SAXOutputter.characters ['>']
[2012-07-10 06:14:08,063] [ICOG802944407]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <SAXOutputter> SAXOutputter.characters ['666642']
[2012-07-10 06:14:08,063] [ICOG802944407]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <SAXOutputter> SAXOutputter.characters ['<']
[2012-07-10 06:14:08,063] [ICOG802944407]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <SAXOutputter> SAXOutputter.characters ['/lSO']
'weblogic.kernel.Default (self-tuning)'> <JMSEventRouter> This message has been tried for first time
[2012-07-10 06:14:07,878] [ICOG802937073]:VideoResponse:Warning [DEBUG] <[STUCK] ExecuteThread: '95' for queue: 'weblogic.kernel.Default (self-tuning)'> <NSStack> NSPop (32)
[2012-07-10 06:14:07,877] [ICOG802944396]:DataResponse:Accepted [DEBUG] <[STUCK] ExecuteThread: '119' for queue: 'weblogic.kernel.Default (self-tuning)'> <SOAPPart> Setting current message form to: FORM_OPTIMIZED (currentMessage is now org.apache.axis.utils.ByteArray)
[2012-07-10 06:14:07,877] [ICOG802944401]:FiosVoiceResponse:Accepted [DEBUG] <[ACTIVE] ExecuteThread: '103' for queue: 'weblogic.kernel.Default (self-tuning)'> <PropertyManager> timestamp calculation, currentTimeStamp : 2012-07-10 nextRefreshTime : 2012-07-10
[2012-07-10 06:14:08,077] [ICOG802941112-01]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <SSPBundleEventDAO> It's a Supp Cancle Order
[2012-07-10 06:14:08,077] [ICOG802941112-01]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <SSPBundleEventDAO> Only Bundle Cancel Exists..
[2012-07-10 06:14:08,077] [ICOG802941112-01]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <SSPBundleEventDAO> isFVBundleCancel true
[2012-07-10 06:14:08,077] [ICOG802941112-01]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <BusinessDelegate> Got Product with productTypeValue as NA
[2012-07-10 06:14:08,077] [ICOG802941112-01]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <BusinessDelegate> Got Product with productTypeValue as FDVBUNDLE
[2012-07-10 06:14:08,077] [ICOG802941112-01]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <BusinessDelegate> Got FiOSVoice Product.. Going to insert int Master table..
[2012-07-10 06:14:08,077] [ICOG802941112-01]:FirmSaleOrder: [DEBUG] <[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <BusinessDelegate> orderType Install
We are using - Weblogic - 11g, Solaris - 5.10, jdk160_31_64
Can some one please suggest
Regards,
Uday

Answer 1

Hi,
These are not an issue but you need couple of tuning on server side.
Make sure to increase your Max permsize to 1024m in case of 64 bit or 512m in case of 32 bit.
Try to increase chunk size and chunkpool size to 8k.
Enable tunneling on your server.
Make your min and max heap size as equal.
try to changes these settings and still you see the issue let me know.
Regards,
Kal