WL Server Stuck Thread on Network Read

Question

I am experiencing an issue with the Weblogic deployment at my organisation.
Firstly an introduction; My Application works by receiving a Web Service (SOAP) Request from a client, the Web Service request is then processed and a internal message packet is generated and sent to a C++ Application ( which is listening on a particular port). The C++ application then processes the information, and returns the response to the web service Layer, which then process the Web Service request back to the calling client.
Currently we are having a situation where some packets are sent to the C++ application as expected. But then we are having situations where some packets NEVER reach the C++ side and the calling client time outs the entire request ( configured timeout is set for 180 seconds) and on the WebLogic Layer we have noticed a significant amount of Stuck ( and Hogging ) Threads. It should be noted that even after a stuck thread is recorded some requests are still being processed normally.
A deeper investigation into the Weblogic Thread dump shows the Weblogic thread is stuck at:
_at sun/nio/ch/IOUtil.read(Ljava/io/FileDescriptor;Ljava/nio/ByteBuffer;JLsun/nio/ch/NativeDispatcher;Ljava/lang/Object;)I(IOUtil.java)[inlined]_+
_at sun/nio/ch/SocketChannelImpl.read(SocketChannelImpl.java:236)[optimized]_+
+_^-- Holding lock: java/lang/Object@0x1a27db60[thin lock]_+
_at com/reso/number/service/comm/SoaSocket.receive(SoaSocket.java:128)[optimized]_+
_at com/reso/number/service/event/OTEvent.submitEvent(TEvent.java:219)[optimized]_+
_at com/reso/number/service/event/SIMQueryEvent.process(QQEvent.java:368)[optimized]_+
It seems that a write is sent to the C++ Application and the thread is 'WAITING' for the response.
Now, the problem is that due to the Infrastructure in place and architectural setup ( We are using Solaris Zone for Node deployment and we have the WebLogic AS on the same node/zone as the C++ application) we are unable to do a network trace to determine whether WebLogic actually sends the request.
Now my questions: i) Are there any known ways to trace the networks packets sent by Java ( maybe via the JVM) ii) Could Garbage collection cause the symptoms being observed? iii)Any recommended ways to resolve this kind of problem?
Thanks for all the help.

Answer 1

Increasing the Stuck Thread Max Time will not help. " Stuck Thread Max Time" tells us, after how much amount of time the thread should mark as stuck. Because by default it is 600 sec, any thread which takes more than 600sec for execution will mark as stuck. If you mention 700 sec, after 700sec the thread is marked as stuck. Even though the thread is marked stuck it still process.
Here we need to find why the thread is stuck at
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-15 "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
sun.nio.ch.SocketDispatcher.read1(SocketDispatcher.java:???)
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:46)
sun.nio.ch.IOUtil.read(IOUtil.java:228)
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:159)
^-- Holding lock: java.lang.Object@16a47a8b[thin lock]
weblogic.socket.NIOSocketMuxer$NIOInputStream.read(NIOSocketMuxer.java:505)
weblogic.servlet.internal.PostInputStream.read(PostInputStream.java:156)
weblogic.servlet.internal.ServletInputStreamImpl.read(ServletInputStreamImpl.java:228)
weblogic.servlet.internal.ServletRequestImpl$RequestParameters.mergePostParams(ServletRequestImpl.java:2062)
You may have to increase the buffer size at sun.nio.