WLS Admin and managed server startup issue

Question

Hi ,
In one of WLS Domain both the Admin Server and Managed Server startup is just hanging at the below mentioned point :
<Security> <BEA-090082> <Security initializing using security realm myrealm.>
After this there are no error messages or logs being generated and the server start up is not proceeding .
Only message shown a while after this is :
Warning : domain runtime mbean server has not been found after trying for 600 seconds
Please provide some inputs on how to troubleshoot this issue .
Checked the AdminServer.log and there are no errors reported there as well .
Some more data from AdminServer.log :
<[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1336448845368> <BEA-090516> <The Authenticator provider has preexisting LDAP data.>
<[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1336448845632> <BEA-090516> <The Authorizer provider has preexisting LDAP data.>
<[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1336448846193> <BEA-090516> <The CredentialMapper provider has preexisting LDAP data.>
<[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1336448846208> <BEA-090516> <The RoleMapper provider has preexisting LDAP data.>
<[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1336448846433> <BEA-090093> <No pre-WLS 8.1 Keystore providers are configured for server AdminServer for security realm myrealm.>
<[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1336448846433> <BEA-090082> <Security initializing using security realm myrealm.>
Thanks
Srinath
Edited by: Srinath Menon on May 8, 2012 9:36 AM

Answer 1

Hi Ashish ,
Thanks for the inputs.
Tried the same steps as mentioned but the issue still remains like that .
Posting the thread dump collected for AdminServer:
===== FULL THREAD DUMP ===============
Tue May 8 10:04:58 2012
Oracle JRockit(R) R28.1.3-11-141760-1.6.0_24-20110301-1430-linux-ia32
"Main Thread" id=1 idx=0x4 tid=9341 prio=5 alive, waiting, native_blocked
-- Waiting for notification on: weblogic/t3/srvr/SubsystemRequest@0x9f81d3c0[fat lock]
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at syncWaitForSignalNoTransition+81(synchronization.c:28)@0xb7ee7c62
at syncWaitForSignal+157(synchronization.c:85)@0xb7ee7dbe
at syncWaitForJavaSignal+53(synchronization.c:93)@0xb7ee7ea6
at RJNI_jrockit_vm_Threads_waitForNotifySignal+53(rnithreads.c:72)@0xb7eb78a6
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/t3/srvr/SubsystemRequest.rendezvouz(SubsystemRequest.java:100)
^-- Lock released while waiting: weblogic/t3/srvr/SubsystemRequest@0x9f81d3c0[fat lock]
at weblogic/t3/srvr/SubsystemRequest.action(SubsystemRequest.java:81)
at weblogic/t3/srvr/SubsystemRequest.start(SubsystemRequest.java:51)
at weblogic/t3/srvr/ServerServicesManager.startService(ServerServicesManager.java:459)
at weblogic/t3/srvr/ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
^-- Holding lock: java/lang/Class@0x9fa95598[biased lock]
at weblogic/t3/srvr/T3Srvr.initializeStandby(T3Srvr.java:879)
at weblogic/t3/srvr/T3Srvr.startup(T3Srvr.java:566)
at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:467)
at weblogic/Server.main(Server.java:71)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x8 tid=9342 prio=5 alive, native_blocked, daemon
at <unknown>(???.c)@0xb7fe9410
at sigiSignalHandlerThread+135(signalhandler.c:131)@0xb7ec2fd8
at thread_stub+146(lifecycle.c:808)@0xb7e2c333
at start_thread+225(:0)@0x515832
at __clone+93(:0)@0x46ae0e
"(OC Main Thread)" id=3 idx=0xc tid=9343 prio=5 alive, native_waiting, daemon
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at syncWaitForSignalNoTransition+81(synchronization.c:28)@0xb7ee7c62
at innerNativeDoWait+894(nativelock.c:614)@0xb7e78caf
at nativeWait+71(nativelock.c:721)@0xb7e79008
at ocMainThread+663(gcthreads.c:581)@0xb7d8a098
at thread_stub+146(lifecycle.c:808)@0xb7e2c333
at start_thread+225(:0)@0x515832
at __clone+93(:0)@0x46ae0e
"(GC Worker Thread 1)" id=? idx=0x10 tid=9344 prio=5 alive, daemon
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at eventTimedWaitNoTransition+62(event.c:72)@0xb7d6bb3f
at eventWaitNoTransition+32(event.c:62)@0xb7d6bc31
at mmGCWorkerThread+134(gcthreads.c:797)@0xb7d89d67
at thread_stub+146(lifecycle.c:808)@0xb7e2c333
at start_thread+225(:0)@0x515832
at __clone+93(:0)@0x46ae0e
"(GC Worker Thread 2)" id=? idx=0x14 tid=9345 prio=5 alive, daemon
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at eventTimedWaitNoTransition+62(event.c:72)@0xb7d6bb3f
at eventWaitNoTransition+32(event.c:62)@0xb7d6bc31
at mmGCWorkerThread+134(gcthreads.c:797)@0xb7d89d67
at thread_stub+146(lifecycle.c:808)@0xb7e2c333
at start_thread+225(:0)@0x515832
at __clone+93(:0)@0x46ae0e
"(Code Generation Thread 1)" id=4 idx=0x18 tid=9348 prio=5 alive, native_waiting, daemon
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at syncWaitForSignalNoTransition+81(synchronization.c:28)@0xb7ee7c62
at innerNativeDoWait+894(nativelock.c:614)@0xb7e78caf
at nativeWait+71(nativelock.c:721)@0xb7e79008
at cg_thread+380(compilerqueue.c:203)@0xb7d4be5d
at thread_stub+146(lifecycle.c:808)@0xb7e2c333
at start_thread+225(:0)@0x515832
at __clone+93(:0)@0x46ae0e
"(Code Optimization Thread 1)" id=5 idx=0x1c tid=9349 prio=5 alive, native_waiting, daemon
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at syncWaitForSignalNoTransition+81(synchronization.c:28)@0xb7ee7c62
at innerNativeDoWait+894(nativelock.c:614)@0xb7e78caf
at nativeWait+71(nativelock.c:721)@0xb7e79008
at cg_thread+380(compilerqueue.c:203)@0xb7d4be5d
at thread_stub+146(lifecycle.c:808)@0xb7e2c333
at start_thread+225(:0)@0x515832
at __clone+93(:0)@0x46ae0e
"(VM Periodic Task)" id=6 idx=0x20 tid=9350 prio=10 alive, native_blocked, daemon
at <unknown>(???.c)@0xb7fe9410
at block_for_safepoint+106(safepoint.c:34)@0xb7ebad5b
at eventTimedWait+239(event.c:83)@0xb7d6bd30
at periodic_task_thread+627(periodictask.c:367)@0xb7e90dd4
at thread_stub+146(lifecycle.c:808)@0xb7e2c333
at start_thread+225(:0)@0x515832
at __clone+93(:0)@0x46ae0e
"Finalizer" id=7 idx=0x24 tid=9351 prio=8 alive, native_waiting, daemon
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at syncWaitForSignalNoTransition+81(synchronization.c:28)@0xb7ee7c62
at innerNativeDoWait+894(nativelock.c:614)@0xb7e78caf
at nativeWait+71(nativelock.c:721)@0xb7e79008
at RJNI_jrockit_memory_Finalizer_waitForFinalizees+211(finalhandles.c:1007)@0xb7d751f4
at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
    at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
    at jrockit/memory/Finalizer$4.run(Finalizer.java:189)
    at java/lang/Thread.run(Thread.java:662)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
"Reference Handler" id=8 idx=0x28 tid=9352 prio=10 alive, native_waiting, daemon
    at <unknown>(???.c)@0xb7fe9410
    at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
    at syncWaitForSignalNoTransition+81(synchronization.c:28)@0xb7ee7c62
    at innerNativeDoWait+894(nativelock.c:614)@0xb7e78caf
    at nativeUnlockAndWait+71(nativelock.c:686)@0xb7e790a8
    at RJNI_java_lang_ref_Reference_waitForActivatedQueue+38(semiref.c:342)@0xb7ebd9b7
    at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method)
    at java/lang/ref/Reference.access$100(Reference.java:11)
    at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
"(Sensor Event Thread)" id=9 idx=0x2c tid=9353 prio=5 alive, native_blocked, daemon
    at <unknown>(???.c)@0xb7fe9410
    at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
    at eventTimedWaitNoTransition+62(event.c:72)@0xb7d6bb3f
    at eventTimedWait+138(event.c:56)@0xb7d6bccb
    at eventWait+32(event.c:50)@0xb7d6bd81
    at sensor_event_loop+37(sensor.c:68)@0xb7ebfda6
    at thread_stub+146(lifecycle.c:808)@0xb7e2c333
    at start_thread+225(:0)@0x515832
    at __clone+93(:0)@0x46ae0e
"VM JFR Buffer Thread" id=10 idx=0x30 tid=9354 prio=5 alive, in native, daemon
    at <unknown>(???.c)@0xb7fe9410
    at bufferthread+2135(vmjfr.c:1008)@0xb71744b8
    at run_agent_thread+128(jvmtithreads.c:524)@0xb7e1f361
    at thread_stub+146(lifecycle.c:808)@0xb7e2c333
    at start_thread+225(:0)@0x515832
    at __clone+93(:0)@0x46ae0e
"Timer-0" id=13 idx=0x34 tid=9358 prio=5 alive, waiting, native_blocked, daemon
    -- Waiting for notification on: java/util/TaskQueue@0x9f9fc350[fat lock]
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at syncWaitForSignalNoTransition+81(synchronization.c:28)@0xb7ee7c62
at syncWaitForSignal+157(synchronization.c:85)@0xb7ee7dbe
at syncWaitForJavaSignal+53(synchronization.c:93)@0xb7ee7ea6
at RJNI_jrockit_vm_Threads_waitForNotifySignal+53(rnithreads.c:72)@0xb7eb78a6
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at java/util/TimerThread.mainLoop(Timer.java:483)
^-- Lock released while waiting: java/util/TaskQueue@0x9f9fc350[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Timer-1" id=14 idx=0x38 tid=9363 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: java/util/TaskQueue@0xa20ee240[fat lock]
at <unknown>(???.c)@0xb7fe9410
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop(Timer.java:509)
^-- Lock released while waiting: java/util/TaskQueue@0xa20ee240[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x3c tid=9364 prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: netscape/ldap/LDAPResponseListener@0x9724f0c8[fat lock]
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at syncWaitForSignalNoTransition+81(synchronization.c:28)@0xb7ee7c62
at syncWaitForSignal+157(synchronization.c:85)@0xb7ee7dbe
at syncWaitForJavaSignal+53(synchronization.c:93)@0xb7ee7ea6
at RJNI_jrockit_vm_Threads_waitForNotifySignal+53(rnithreads.c:72)@0xb7eb78a6
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at netscape/ldap/LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:200)
^-- Lock released while waiting: netscape/ldap/LDAPResponseListener@0x9724f0c8[recursive]
at netscape/ldap/LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:101)
^-- Lock released while waiting: netscape/ldap/LDAPResponseListener@0x9724f0c8[fat lock]
at netscape/ldap/LDAPConnection.sendRequest(LDAPConnection.java:1796)
^-- Holding lock: netscape/ldap/LDAPConnection@0x9721f310[biased lock]
at netscape/ldap/LDAPConnection.simpleBind(LDAPConnection.java:1763)
at netscape/ldap/LDAPConnection.authenticate(LDAPConnection.java:1264)
at netscape/ldap/LDAPConnection.authenticate(LDAPConnection.java:1273)
at netscape/ldap/LDAPConnection.bind(LDAPConnection.java:1562)
at weblogic/security/providers/authentication/LDAPAtnDelegate$LDAPFactory.newInstance(LDAPAtnDelegate.java:4134)
at weblogic/security/utils/Pool.newInstance(Pool.java:37)
at weblogic/security/utils/Pool.getInstance(Pool.java:33)
at weblogic/security/providers/authentication/LDAPAtnDelegate.getConnection(LDAPAtnDelegate.java:3478)
at weblogic/security/providers/authentication/LDAPAtnDelegate.getConnection(LDAPAtnDelegate.java:3470)
at weblogic/security/providers/authentication/LDAPAtnDelegate.authenticate(LDAPAtnDelegate.java:3522)
at weblogic/security/providers/authentication/LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:227)
at com/bea/common/security/internal/service/LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)
at jrockit/vm/AccessController.doPrivileged(AccessController.java:268)
at com/bea/common/security/internal/service/LoginModuleWrapper.login(LoginModuleWrapper.java:106)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
at jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
    at sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
    at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java/lang/reflect/Method.invoke(Method.java:597)
    at javax/security/auth/login/LoginContext.invoke(LoginContext.java:769)
    at javax/security/auth/login/LoginContext.access$000(LoginContext.java:186)
    at javax/security/auth/login/LoginContext$4.run(LoginContext.java:683)
    at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)
    at jrockit/vm/AccessController.doPrivileged(AccessController.java:268)
    at javax/security/auth/login/LoginContext.invokePriv(LoginContext.java:680)
    at javax/security/auth/login/LoginContext.login(LoginContext.java:579)
    at com/bea/common/security/internal/service/JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:113)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    at jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
    at sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
    at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java/lang/reflect/Method.invoke(Method.java:597)
    at com/bea/common/security/internal/utils/Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
    at $Proxy28.login(Ljava/lang/String;Ljavax/security/auth/callback/CallbackHandler;Lweblogic/security/service/ContextHandler;)Lcom/bea/common/security/service/Identity;(Unknown Source)
    at weblogic/security/service/internal/WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:89)
    at com/bea/common/security/internal/service/JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:82)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    at jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
    at sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
    at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java/lang/reflect/Method.invoke(Method.java:597)
    at com/bea/common/security/internal/utils/Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
    at $Proxy46.authenticate(Ljavax/security/auth/callback/CallbackHandler;Lweblogic/security/service/ContextHandler;)Lcom/bea/common/security/service/Identity;(Unknown Source)
    at weblogic/security/service/WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:40)
    at weblogic/security/service/PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:348)
    at weblogic/security/service/CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:929)
    at weblogic/security/service/CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1050)
    at weblogic/security/service/SecurityServiceManager.initialize(SecurityServiceManager.java:873)
    at weblogic/security/SecurityService.start(SecurityService.java:141)
    at weblogic/t3/srvr/SubsystemRequest.run(SubsystemRequest.java:64)
    at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
"JFR request timer" id=16 idx=0x40 tid=9365 prio=5 alive, waiting, native_blocked, daemon
    -- Waiting for notification on: java/util/TaskQueue@0xa2049280[fat lock]
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at syncWaitForSignalNoTransition+81(synchronization.c:28)@0xb7ee7c62
at syncWaitForSignal+157(synchronization.c:85)@0xb7ee7dbe
at syncWaitForJavaSignal+53(synchronization.c:93)@0xb7ee7ea6
at RJNI_jrockit_vm_Threads_waitForNotifySignal+53(rnithreads.c:72)@0xb7eb78a6
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Object.wait(Object.java:485)
at java/util/TimerThread.mainLoop(Timer.java:483)
^-- Lock released while waiting: java/util/TaskQueue@0xa2049280[fat lock]
at java/util/TimerThread.run(Timer.java:462)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"weblogic.time.TimeEventGenerator" id=18 idx=0x44 tid=9366 prio=9 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/time/common/internal/TimeTable@0xa2089c70[fat lock]
at <unknown>(???.c)@0xb7fe9410
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0xa2089c70[fat lock]
at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"JMAPI event thread" id=19 idx=0x48 tid=9367 prio=5 alive, in native, daemon
at <unknown>(???.c)@0xb7fe9410
at eventTimedWaitNoTransitionImpl+79(event.c:90)@0xb7d6bae0
at syncWaitForSignalNoTransition+81(synchronization.c:28)@0xb7ee7c62
at innerNativeDoWait+526(nativelock.c:614)@0xb7e78b3f
at nlMonitorWaitInner+79(nativelock.c:1112)@0xb7e78df0
at nlMonitorWaitNoTransition+38(nativelock.c:1132)@0xb7e78e47
at nativeRawMonitorWait+78(rawmonitor.c:161)@0xb7ea8caf
at jvmti_RawMonitorWait+111(jvmtirawmonitors.c:95)@0xb7e1b420
at threadProc+291(jmapi.c:305)@0x90452bd4
at run_agent_thread+128(jvmtithreads.c:524)@0xb7e1f361
at thread_stub+146(lifecycle.c:808)@0xb7e2c333
at start_thread+225(:0)@0x515832
at __clone+93(:0)@0x46ae0e
"weblogic.timers.TimerThread" id=20 idx=0x4c tid=9368 prio=9 alive, waiting, native_blocked, daemon
-- Waiting for notification on: weblogic/timers/internal/TimerThread@0xa203dbf0[fat lock]
at <unknown>(???.c)@0xb7fe9410
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:262)
^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0xa203dbf0[fat lock]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace