Performance Issues in weblogic managed server

Question

Hi,
We are facing performance issues in our managed servers in weblogic domain and in logs it is showing this error continuously.
<Oct 2, 2013 10:39:27 AM MST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '47' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "700" seconds working on the request "Workmanager: OIMMDBWorkManager, Version: 0, Scheduled=false, Started=true, Started time: 700360 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-294 "[STUCK] ExecuteThread: '47' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
    -- Blocked trying to get lock: java.io.PrintStream@41a53700[not locked]
    java.io.PrintStream.println(PrintStream.java:755)
    com.wyndham.operations.user.eventHandlers.PostCreateOperationsEventHandler.execute(PostCreateOperationsEventHandler.java:60)
    oracle.iam.platform.kernel.impl.OrchProcessData.runPostProcessEvents(OrchProcessData.java:1225)
    oracle.iam.platform.kernel.impl.OrchProcessData.runEvents(OrchProcessData.java:665)
    oracle.iam.platform.kernel.impl.OrchProcessData.executeEvents(OrchProcessData.java:268)
    oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:810)
    oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.notifyParentProcess(OrchestrationEngineImpl.java:971)
    oracle.iam.platform.kernel.impl.OrchProcessData.runEvents(OrchProcessData.java:665)
    oracle.iam.platform.kernel.impl.OrchProcessData.executeEvents(OrchProcessData.java:268)
    oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:810)
    oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeChildProcess(OrchestrationEngineImpl.java:880)
    oracle.iam.platform.kernel.impl.OrchProcessData.handleAdditionalChanges(OrchProcessData.java:552)
    oracle.iam.platform.kernel.impl.OrchProcessData.runEvents(OrchProcessData.java:665)
    oracle.iam.platform.kernel.impl.OrchProcessData.executeEvents(OrchProcessData.java:268)
    oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:815)
    oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:843)
    oracle.iam.platform.kernel.impl.OrhestrationAsyncTask.execute(OrhestrationAsyncTask.java:97)
    oracle.iam.platform.async.impl.TaskExecutor.executeUnmanagedTask(TaskExecutor.java:99)
    oracle.iam.platform.async.impl.TaskExecutor.execute(TaskExecutor.java:52)
    oracle.iam.platform.async.messaging.MessageReceiver.onMessage(MessageReceiver.java:60)
    sun.reflect.GeneratedMethodAccessor1493.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:575)
    com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)
    com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
    com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
    $Proxy460.onMessage(Unknown Source)
    weblogic.ejb.container.internal.MDListener.execute(MDListener.java:518)
    weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:467)
    weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
    weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4638)
    weblogic.jms.client.JMSSession.execute(JMSSession.java:4134)
    weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3820)
    weblogic.jms.client.JMSSession.access$000(JMSSession.java:115)
We have performed workload manager tuning in weblogic console as below.
The property ‘Maximum Threads Constraint’ of work managers OIMMDBWorkManager = 80 and OIMUIWorkManager = 20
Please suggest the solution.

Answer 1

Hi,
If you look more closely at the STUCK Thread stack trace, you will notice that the thread is attempting to acquire a LOCK on the following Object monitor: java.io.PrintStream@41a53700. The native thread state is also in a BLOCKED state.
My primary recommendation is to generate a full JVM Thread Dump (next re-occurence) so we can identify the potential culprit Thread holding such lock. One more thing, please see if you have multiple snapshots of this thread 294?
I also recommend to review the following source code:
com.wyndham.operations.user.eventHandlers.PostCreateOperationsEventHandler.execute(PostCreateOperationsEventHandler.java:60)
Sharing the same Stream object in this case would trigger thread lock contention given java.io.PrintStream has built-in synchronization ex:
   public void println(String x) {
  synchronized (this) {
     print(x);
     newLine();
Regards,
P-H
http://javaeesupportpatterns.blogspot.com