Weblogic 8.14: Stuck Threads during JMS - IO native write operations

Question

G'day!
Hope somebody can advise a solution on the following...
Recently message loads have started to increase on our weblogic 8.14 cluster (3 servers) and we started to see an almost daily occurrence of Stuck Thread issues. From thread dumps we could see these were caused by jrockit/net/SocketNativeIO.write processes. We concluded we had reached the max performance limit for the file based JMS stores and so we moved these to database based JMS stores. This certainly seemed to improve matters - but we have recently seen the issue return. Which is puzzling, as technically we thought all messages would be persisted to the database, and any file IO on the application server would be eliminated. Our database resides on a separate dedicated oracle server. We tracked a recent cause to a large message (200K) which caused the Struck Thread - resulting in a build up of pending messages. After bouncing the server the message processed ok. Any thoughts on how to overcome these issues - greatly appreciated. Here's some detail on the stuck thread from the thread dump: (+Why does weblogic use IO when the JMS store is on a database?+). All messages are persistant.
"ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" prio=5 id=0xc00 pid=27544 active, daemon
at jrockit/net/SocketNativeIO.write(Native Method)@0xf3aeae10
at jrockit/net/SocketNativeIO.write(Unknown Source)@0xf3aeae9c
at jrockit/io/NativeIO.write(Unknown Source)@0xf3ae62c6
at java/net/AbstractSocketImpl$2.write(Unknown Source)@0xf3aeada4
at jrockit/io/NativeIOOutputStream.write(Unknown Source)@0xf3ae61ec
at jrockit/io/NativeIOOutputStream.write(Unknown Source)@0xf3ae618d
at java/io/DataOutputStream.write(Unknown Source)@0xf4e3ec25
^-- Holding lock: java/io/DataOutputStream@0x4309fc80[thin lock]
at com/ibm/mq/MQInternalCommunications.send(MQInternalCommunications.java:1022)@0xed34364a
^-- Holding lock: com/ibm/mq/MQInternalCommunications@0x4309f7e0[thin lock]
at com/ibm/mq/MQSESSIONClient.lowLevelComms(MQSESSIONClient.java:2832)@0xed342fbd
^-- Holding lock: java/lang/Integer@0x4309f8a0[thin lock]
at com/ibm/mq/MQSESSIONClient.MQPUT(MQSESSIONClient.java:3844)@0xed34271b
at com/ibm/mq/MQQueue.putMsg2(MQQueue.java:1486)@0xed342262
^-- Holding lock: com/ibm/mq/MQSPIQueue@0x4312ce80[thin lock]
at com/ibm/mq/jms/MQMessageProducer.sendInternal(MQMessageProducer.java:1569)@0xed33d1ab
at com/ibm/mq/jms/MQMessageProducer.send(MQMessageProducer.java:1012)@0xed860a49
at com/ibm/mq/jms/MQMessageProducer.send(MQMessageProducer.java:1046)@0xed8605f4
at weblogic/jms/adapter/JMSBaseConnection.sendInternal(JMSBaseConnection.java:667)@0xef7df0bf
^-- Holding lock: weblogic/jms/adapter/JMSBaseConnection@0x43093748[thin lock]
at weblogic/jms/adapter/JMSBaseConnection.access$200(JMSBaseConnection.java:80)@0xef7dee28
at weblogic/jms/adapter/JMSBaseConnection$6.run(JMSBaseConnection.java:647)@0xef7dee1d
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)@0xf2fcbb88
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:147)@0xf2fcbb0a
at weblogic/jms/adapter/JMSBaseConnection.send(JMSBaseConnection.java:644)@0xef7decf3
at weblogic/jms/adapter/JMSConnectionHandle.send(JMSConnectionHandle.java:144)@0xef7dec7c
at jrockit/reflect/NativeMethodInvoker.invoke0(Native Method)@0xf4f10d10
at jrockit/reflect/NativeMethodInvoker.invoke(Unknown Source)@0xf4f10e98
at jrockit/reflect/VirtualNativeMethodInvoker.invoke(Unknown Source)@0xf4e86d70
at java/lang/reflect/Method.invoke(Unknown Source)@0xf4f104c3
at weblogic/connector/common/internal/ConnectionWrapper.invoke(ConnectionWrapper.java:149)@0xed96a498
at $Proxy14.send(Unknown Source)@0xef7dec04
at weblogic/jms/bridge/internal/MessagingBridge.onMessageInternal(MessagingBridge.java:1258)@0xed857b25
at weblogic/jms/bridge/internal/MessagingBridge.onMessage(MessagingBridge.java:1181)@0xed857864
at weblogic/jms/adapter/JMSBaseConnection$27.run(JMSBaseConnection.java:1943)@0xef7de623
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)@0xf2fcbb88
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:147)@0xf2fcbb0a
at weblogic/jms/adapter/JMSBaseConnection.onMessage(JMSBaseConnection.java:1939)@0xef7de586
at weblogic/jms/client/JMSSession.onMessage(JMSSession.java:2678)@0xeee7e30c
at weblogic/jms/client/JMSSession.execute(JMSSession.java:2598)@0xeee7d25d
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:219)@0xf4e14070
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:178)@0xf4e1fddb
at java/lang/Thread.startThreadFromVM(Unknown Source)@0xf4f3c9f3
Thanks
C.

Answer 1

I'm not sure what operating system you're running, but I only have experience of WLS8.1 on Windows.
We did extensive load testing of JMS before we went live, and found JDBC file stores to be rubbish in comparison to file store. When a message is persisted, it has to be written to the database, then when it is consumed, it must be deleted from the table. This simply killed our system when we load tested it.
One thing we did do though, was use a second JMS server and second file store for one particularly heavy queue, where we did have IO issues. This was with a JMS distributed destination. We also use a synchronous write policy of cache-flush on the file store and we also have configured a paging store to page messages so the app server does not run out of memory.
200K isn't by any means a huge message though, and shouldn't be any problem at all, we have 1mb messages quite regularly.
I notice from the stack trace, that you have MQ series aswell. I'm not experienced with MQ, so can't give any information, but what I would try is the following.
Using a JMS server with JMS File storage, develop a small java client to write as many messages to the queue and effectively stress the JMS subsystem just to establish if the filestore is the problem. This should be done without a connection to MQ and also use large messages. if you don't get any issues and WebLogic JMS with file store looks ok, then attach a consumer to the queue, which should also be a simple java client. Then you have production and consumption going on, which will result in writes and read's/deletes from the jms file store.
Once you've stress tested JMS and you are satisfied that its ok or not, then add the MQseries connection and see how that affects things.
You may well have already done all this, so please feel free to ignore me! I did a load of this kind of stuff before we went live to try and bash JMS to bits, but I didn't manage it! The only thing we did was add a second file store and second JMS server.
I just looked back in my support cases, and found we'd had a similar issue - disk fragmentation seemed to be one of the factors here too. The JMS file store was heavily fragmented. That's windows for you though!!!
Hope that helps.
Pete