WLS 10.0 cannot get Admin Console to start

Question

Hi,
We have WLS 10.0 with Jrocket on a Windows Server.
We have two domains and they have been working fine for well over a year until this morning. I cannot bring up the Admin Console for the 2 domains. The domains are setup to start up automatically and they get stopped and started every Saturday. This week the restart seems to have failed, The services were showing as started but we could not login into the WLS Admin Console or the web applications that the domains host.
As far as i know nothing should have changed with the server.
Event viewer is not showing any issues.
I tried to start the WLS for the 'da' domain from the cmd line and am now seeing the following in the AdminServer.log (FYI, the domain log is not being created). I have had to remove some lines to make the post :
========================================================================
####<Feb 14, 2010 2:20:15 PM GST> <Notice> <WebLogicServer> <ePIMSEDMS2> <> <Main Thread> <> <> <> <1266142815354> <BEA-000395> <Following extensions directory contents added to the end of the classpath:
D:\bea10\wlserver_10.0\platform\lib\p13n\p13n-schemas.jar;D:\bea10\wlserver_10.0\platform\lib\p13n\p13n_common.jar;D:\bea10\wlserver_10.0\platform\lib\p13n\p13n_system.jar;D:\bea10\wlserver_10.0\platform\lib\wlp\netuix_common.jar;D:\bea10\wlserver_10.0\platform\lib\wlp\netuix_schemas.jar;D:\bea10\wlserver_10.0\platform\lib\wlp\netuix_system.jar;D:\bea10\wlserver_10.0\platform\lib\wlp\wsrp-client.jar;D:\bea10\wlserver_10.0\platform\lib\wlp\wsrp-common.jar>
####<Feb 14, 2010 2:20:15 PM GST> <Info> <WebLogicServer> <ePIMSEDMS2> <> <Main Thread> <> <> <> <1266142815870> <BEA-000377> <Starting WebLogic Server with BEA JRockit(R) Version R26.4.0-63_CR302700-72606-1.5.0_06-20061127-1108-win-ia32 from BEA Systems, Inc.>
####<Feb 14, 2010 2:20:16 PM GST> <Info> <Management> <ePIMSEDMS2> <> <Main Thread> <> <> <> <1266142816183> <BEA-141107> <Version: WebLogic Server 10.0 Wed May 9 18:10:27 EDT 2007 933139 >
####<Feb 14, 2010 2:20:17 PM GST> <Info> <WebLogicServer> <ePIMSEDMS2> <> <Main Thread> <> <> <> <1266142817511> <BEA-000215> <Loaded License : D:\bea10\license.bea>
####<Feb 14, 2010 2:20:17 PM GST> <Notice> <WebLogicServer> <ePIMSEDMS2> <> <Main Thread> <> <> <> <1266142817526> <BEA-000365> <Server state changed to STARTING>
####<Feb 14, 2010 2:20:23 PM GST> <Notice> <WebLogicServer> <ePIMSEDMS2> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1266142823995> <BEA-000365> <Server state changed to STANDBY>
####<Feb 14, 2010 2:20:24 PM GST> <Notice> <WebLogicServer> <ePIMSEDMS2> <AdminServer> <Main Thread> <<WLS Kernel>> <> <> <1266142824011> <BEA-000365> <Server state changed to STARTING>
####<Feb 14, 2010 2:20:24 PM GST> <Info> <Deployer> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1266142824026> <BEA-149209> <Resuming.>
####<Feb 14, 2010 2:20:24 PM GST> <Info> <WorkManager> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1266142824183> <BEA-002901> <Creating WorkManager "consoleWorkManager" for module "null" and application "consoleapp">
####<Feb 14, 2010 2:20:24 PM GST> <Info> <J2EE> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1266142824292> <BEA-160192> <While upgrading weblogic.xml, encountered "debug" param "jsp-param". This param is unknown and will be removed>
####<Feb 14, 2010 2:20:24 PM GST> <Info> <J2EE> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1266142824698> <BEA-160192> <While upgrading weblogic.xml, encountered "debug" param "jsp-param". This param is unknown and will be removed>
####<Feb 14, 2010 2:20:25 PM GST> <Info> <Deployer> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1266142825011> <BEA-149059> <Module da.war of application da is transitioning from STATE_NEW to STATE_PREPARED on server AdminServer.>
####<Feb 14, 2010 2:20:25 PM GST> <Info> <Deployer> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1266142825354> <BEA-149060> <Module da.war of application da successfully transitioned from STATE_NEW to STATE_PREPARED on server AdminServer.>
####<Feb 14, 2010 2:20:25 PM GST> <Info> <Deployer> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1266142825370> <BEA-149059> <Module da.war of application da is transitioning from STATE_PREPARED to STATE_ADMIN on server AdminServer.>
####<Feb 14, 2010 2:20:25 PM GST> <Info> <Deployer> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1266142825370> <BEA-149060> <Module da.war of application da successfully transitioned from STATE_PREPARED to STATE_ADMIN on server AdminServer.>
####<Feb 14, 2010 2:20:37 PM GST> <Info> <netuix> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1266142837729> <BEA-423101> <[console] Initializing the NetUIx container>
####<Feb 14, 2010 2:20:39 PM GST> <Info> <netuix> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1266142839433> <BEA-423120> <WEB-INF/client-classifications.xml file not found at uri [webapp]. Classifications functionality not enabled.>
####<Feb 14, 2010 2:36:43 PM GST> <Notice> <WebLogicServer> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <> <> <1266143803604> <BEA-000388> <JVM called WLS shutdown hook. The server will force shutdown now>
####<Feb 14, 2010 2:36:43 PM GST> <Alert> <WebLogicServer> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <> <> <1266143803620> <BEA-000396> <Server shutdown has been requested by <WLS Kernel>>
####<Feb 14, 2010 2:36:43 PM GST> <Notice> <WebLogicServer> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <> <> <1266143803620> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
####<Feb 14, 2010 2:36:43 PM GST> <Info> <JMX> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <> <> <1266143803683> <BEA-149513> <JMX Connector Server stopped at service:jmx:iiop://10.113.202.96:8101/jndi/weblogic.management.mbeanservers.edit .>
####<Feb 14, 2010 2:36:43 PM GST> <Info> <JMX> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <> <> <1266143803683> <BEA-149513> <JMX Connector Server stopped at service:jmx:iiop://10.113.202.96:8101/jndi/weblogic.management.mbeanservers.runtime .>
####<Feb 14, 2010 2:36:43 PM GST> <Info> <WebService> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <> <> <1266143803683> <BEA-220028> <Web Service reliable agents are suspended.>
####<Feb 14, 2010 2:36:43 PM GST> <Info> <WebService> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <> <> <1266143803683> <BEA-220029> <Web Service reliable agents are shut down.>
####<Feb 14, 2010 2:36:43 PM GST> <Info> <SAFService> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <> <> <1266143803698> <BEA-281004> <SAF Service has been suspended.>
####<Feb 14, 2010 2:36:43 PM GST> <Info> <SAFService> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <> <> <1266143803698> <BEA-281005> <SAF Service has been shut down.>
####<Feb 14, 2010 2:36:43 PM GST> <Info> <Deployer> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <> <> <1266143803714> <BEA-149059> <Module da.war of application da is transitioning from STATE_ADMIN to STATE_NEW on server AdminServer.>
####<Feb 14, 2010 2:36:49 PM GST> <Info> <Management> <ePIMSEDMS2> <AdminServer> <ApplicationFilePoller-consoleapp> <<anonymous>> <> <> <1266143809277> <BEA-400411> <ApplicationFilePoller exiting. Further application file updates will not be seen.>
####<Feb 14, 2010 2:37:13 PM GST> <Notice> <WebLogicServer> <ePIMSEDMS2> <AdminServer> <Thread-19> <<WLS Kernel>> <> <> <1266143833624> <BEA-000378> <Server failed to shutdown within the configured timeout of 30 seconds. The server process will exit now.>
####<Feb 14, 2010 2:37:13 PM GST> <Error> <> <ePIMSEDMS2> <AdminServer> <Thread-19> <<WLS Kernel>> <> <> <1266143833640> <BEA-000000> <
===== FULL THREAD DUMP ===============
Sun Feb 14 14:37:13 2010
BEA JRockit(R) R26.4.0-63_CR302700-72606-1.5.0_06-20061127-1108-win-ia32
"Main Thread" id=1 idx=0x2 tid=8140 prio=5 alive, in native, blocked
-- Blocked trying to get lock: java/lang/Class@0x00DB5430[thin lock]
at jrockit/vm/Threads.sleep(I)V(Native Method)
at jrockit/vm/Locks.waitForThinRelease(Ljava/lang/Object;I)I(Unknown Source)
at jrockit/vm/Locks.monitorEnterSecondStage(Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source)
at jrockit/vm/Locks.monitorEnter(Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at java/lang/Shutdown.exit(I)V(Unknown Source)
at java/lang/Runtime.exit(I)V(Runtime.java:90)
at java/lang/System.exit(I)V(Unknown Source)
at weblogic/Server.main([Ljava/lang/String;)V(Server.java:87)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x4 tid=1196 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: weblogic/t3/srvr/T3Srvr$2@0x00DEDF70[fat lock]
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Thread.join(J)V(Unknown Source)
^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr$2@0x00DEDF70[fat lock]
at java/lang/Thread.join()V(Unknown Source)
at java/lang/Shutdown.runHooks()V(Unknown Source)
at java/lang/Shutdown.sequence()V(Unknown Source)
at java/lang/Shutdown.exit(I)V(Unknown Source)
^-- Holding lock: java/lang/Class@0x00DB5430[thin lock]
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace
"(Code Generation Thread 1)" id=3 idx=0x6 tid=1736 prio=5 alive, in native, native_waiting, daemon
"(Code Optimization Thread 1)" id=4 idx=0x8 tid=436 prio=5 alive, in native, native_waiting, daemon
"(GC Main Thread)" id=5 idx=0xa tid=3404 prio=5 alive, in native, daemon
"(GC Worker Thread 1)" id=? idx=0xc tid=7548 prio=5 alive, native_waiting, daemon
"(GC Worker Thread 2)" id=? idx=0xe tid=8164 prio=5 alive, native_waiting, daemon
"(GC Worker Thread 3)" id=? idx=0x10 tid=6716 prio=5 alive, native_waiting, daemon
"(GC Worker Thread 4)" id=? idx=0x12 tid=5452 prio=5 alive, native_waiting, daemon
"(VM Periodic Task)" id=6 idx=0x14 tid=4496 prio=10 alive, in native, daemon
"Timer-0" id=11 idx=0x1a tid=7536 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/TaskQueue@0x00CB5758[fat lock]
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)
at java/lang/Object.wait()V(Native Method)
at java/util/TimerThread.mainLoop()V(Timer.java:483)
^-- Lock released while waiting: java/util/TaskQueue@0x00CB5758[fat lock]
at java/util/TimerThread.run()V(Timer.java:462)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace
"Timer-1" id=12 idx=0x1c tid=5060 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: java/util/TaskQueue@0x04F27760[fat lock]
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/util/TimerThread.mainLoop()V(Timer.java:509)
^-- Lock released while waiting: java/util/TaskQueue@0x04F27760[fat lock]
at java/util/TimerThread.run()V(Timer.java:462)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=13 idx=0x1e tid=732 prio=5 alive, in native, daemon
at sun/nio/ch/SocketDispatcher.read0(Ljava/io/FileDescriptor;JI)I(Native Method)
at sun/nio/ch/SocketDispatcher.read(Ljava/io/FileDescriptor;JI)I(SocketDispatcher.java:25)
at sun/nio/ch/IOUtil.readIntoNativeBuffer(Ljava/io/FileDescriptor;Ljava/nio/ByteBuffer;JLsun/nio/ch/NativeDispatcher;Ljava/lang/Object;)I(IOUtil.java:233)
at sun/nio/ch/IOUtil.read0(Ljava/io/FileDescriptor;Ljava/nio/ByteBuffer;JLsun/nio/ch/NativeDispatcher;Ljava/lang/Object;)I(IOUtil.java:200)
at sun/nio/ch/IOUtil.read(Ljava/io/FileDescriptor;Ljava/nio/ByteBuffer;JLsun/nio/ch/NativeDispatcher;Ljava/lang/Object;)I(Unknown Source)
at sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I(SocketChannelImpl.java:207)
^-- Holding lock: java/lang/Object@0x0707C800[thin lock]
at com/documentum/fc/impl/util/io/MessageChannel.readSocket(Ljava/nio/ByteBuffer;I)V(MessageChannel.java:123)
at com/documentum/fc/impl/util/io/MessageChannel.readLength()I(MessageChannel.java:100)
at com/documentum/fc/impl/util/io/MessageChannel.getIncomingMessageLength()I(MessageChannel.java:92)
at com/documentum/fc/impl/util/io/MessageChannel.read(Ljava/nio/ByteBuffer;)I(MessageChannel.java:77)
^-- Holding lock: com/documentum/fc/impl/util/io/MessageChannel@0x07088B80[thin lock]
at com/documentum/fc/client/impl/connection/netwise/AbstractNetwiseRpcClient.receiveMessage(Ljava/nio/ByteBuffer;)V(AbstractNetwiseRpcClient.java:167)
at com/documentum/fc/client/impl/connection/docbase/netwise/NetwiseDocbaseRpcClient.applyForStringInternal(Ljava/lang/String;Lcom/documentum/fc/common/IDfId;Lcom/documentum/fc/client/impl/typeddata/ITypedData;ZZ)Ljava/lang/String;(NetwiseDocbaseRpcClient.java:746)
^-- Holding lock: com/documentum/fc/client/impl/connection/docbase/netwise/NetwiseDocbaseRpcClient@0x07088C18[recursive]
at com/documentum/fc/client/impl/connection/docbase/netwise/NetwiseDocbaseRpcClient.fetchTypeViaApply(Ljava/lang/String;I)Lcom/documentum/fc/client/impl/typeddata/ILiteType;(NetwiseDocbaseRpcClient.java:894)
at com/documentum/fc/client/impl/connection/docbase/netwise/NetwiseDocbaseRpcClient.fetchType(Ljava/lang/String;I)Lcom/documentum/fc/client/impl/typeddata/ILiteType;(NetwiseDocbaseRpcClient.java:821)
^-- Holding lock: com/documentum/fc/client/impl/connection/docbase/netwise/NetwiseDocbaseRpcClient@0x07088C18[thin lock]
at com/documentum/fc/client/impl/connection/docbase/DocbaseConnection$11.evaluate()Ljava/lang/Object;(DocbaseConnection.java:1309)
^-- Holding lock: com/documentum/fc/client/impl/connection/docbase/DocbaseConnection@0x06E11D48[recursive]
at com/documentum/fc/client/impl/connection/docbase/DocbaseConnection.evaluateRpc(Lcom/documentum/fc/impl/util/Functor;Z)Ljava/lang/Object;(DocbaseConnection.java:1014)
^-- Holding lock: com/documentum/fc/client/impl/connection/docbase/DocbaseConnection@0x06E11D48[recursive]
at com/documentum/fc/client/impl/connection/docbase/DocbaseConnection.fetchType(Ljava/lang/String;Z)Lcom/documentum/fc/client/impl/typeddata/ILiteType;(DocbaseConnection.java:1302)
at com/documentum/fc/client/impl/connection/LiteTypeManager.getRawType(Ljava/lang/String;Lcom/documentum/fc/client/impl/connection/docbase/IDocbaseConnection;)Lcom/documentum/fc/client/impl/typeddata/ILiteType;(LiteTypeManager.java:104)
at com/documentum/fc/client/impl/connection/LiteTypeManager.getType(Ljava/lang/String;Ljava/util/List;Lcom/documentum/fc/client/impl/connection/docbase/IDocbaseConnection;)Lcom/documentum/fc/client/impl/typeddata/ILiteType;(LiteTypeManager.java:78)
at com/documentum/fc/client/impl/connection/LiteTypeManager.getType(Ljava/lang/String;Lcom/documentum/fc/client/impl/connection/docbase/IDocbaseConnection;)Lcom/documentum/fc/client/impl/typeddata/ILiteType;(LiteTypeManager.java:61)
at com/documentum/fc/client/impl/connection/docbase/DocbaseConnection.getLiteType(Ljava/lang/String;)Lcom/documentum/fc/client/impl/typeddata/ILiteType;(DocbaseConnection.java:1421)
^-- Holding lock: com/documentum/fc/client/impl/connection/docbase/DocbaseConnection@0x06E11D48[recursive]
at com/documentum/fc/client/impl/Docbase.typeExists(Lcom/documentum/fc/client/impl/connection/docbase/IDocbaseConnection;Ljava/lang/String;)Z(Docbase.java:143)
at com/documentum/fc/client/impl/Docbase.determineBof2Support(Lcom/documentum/fc/client/impl/connection/docbase/IDocbaseConnection;)V(Docbase.java:133)
at com/documentum/fc/client/impl/Docbase.<init>(Lcom/documentum/fc/client/impl/connection/docbase/IDocbaseConnection;Lcom/documentum/fc/client/impl/Server;)V(Docbase.java:90)
at com/documentum/fc/client/impl/Docbase.getInstance(Lcom/documentum/fc/client/impl/connection/docbase/IDocbaseConnection;Lcom/documentum/fc/client/impl/Server;)Lcom/documentum/fc/client/impl/Docbase;(Docbase.java:58)
^-- Holding lock: java/util/HashMap@0x091A3598[thin lock]
at com/documentum/fc/client/impl/connection/docbase/DocbaseConnection.getDocbase()Lcom/documentum/fc/client/impl/Docbase;(DocbaseConnection.java:1553)
^-- Holding lock: com/documentum/fc/client/impl/connection/docbase/DocbaseConnection@0x06E11D48[thin lock]
at com/documentum/fc/client/impl/session/Session.getDocbase()Lcom/documentum/fc/client/impl/Docbase;(Session.java:3113)
at com/documentum/fc/client/impl/session/Session.getObjectManager(Lcom/documentum/fc/common/IDfId;)Lcom/documentum/fc/client/impl/objectmanager/PersistentObjectManager;(Session.java:775)
at com/documentum/fc/client/impl/session/Session.getObject(Lcom/documentum/fc/common/IDfId;)Lcom/documentum/fc/client/IDfPersistentObject;(Session.java:834)
at com/documentum/fc/client/impl/session/SessionHandle.getObject(Lcom/documentum/fc/common/IDfId;)Lcom/documentum/fc/client/IDfPersistentObject;(SessionHandle.java:580)
at com/documentum/fc/client/security/impl/IpAndRcHelper.getRegistration(Lcom/documentum/fc/client/security/internal/IPublicIdentity;Lcom/documentum/fc/client/IDfSession;)Lcom/documentum/fc/client/privilege/internal/IClientRegistration;(IpAndRcHelper.java:472)
at com/documentum/fc/client/security/impl/DfcIdentityPublisher.getRegistration(Lcom/documentum/fc/client/security/internal/IPublicIdentity;Lcom/documentum/fc/client/IDfSession;)Lcom/documentum/fc/client/privilege/internal/IClientRegistration;(DfcIdentityPublisher.java:469)
at com/documentum/fc/client/security/impl/DfcIdentityPublisher.isRegistration(Lcom/documentum/fc/client/security/internal/IPublicIdentity;Lcom/documentum/fc/client/IDfSession;)Z(DfcIdentityPublisher.java:446)
at com/documentum/fc/client/security/impl/DfcIdentityPublisher.publish(Z)V(DfcIdentityPublisher.java:134)
^-- Holding lock: java/lang/Object@0x03E097F0[thin lock]
at com/documentum/fc/client/security/impl/DfcIdentityPublisher.publish()V(DfcIdentityPublisher.java:102)
at com/documentum/fc/client/security/internal/RegistrationMgr.register()Z(RegistrationMgr.java:34)
at com/documentum/fc/impl/RuntimeContext.<clinit>()V(RuntimeContext.java:191)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
at jrockit/vm/RNI.generateFixedCode(I)I(Native Method)
at com/documentum/fc/impl/RuntimeContext.getInstance()Lcom/documentum/fc/impl/RuntimeContext;(RuntimeContext.java:0)
at com/documentum/fc/client/DfClient.<clinit>()V(DfClient.java:700)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
at jrockit/vm/RNI.generateFixedCode(I)I(Native Method)
at com/documentum/fc/client/DfClient.getLocalClient()Lcom/documentum/fc/client/IDfClient;(DfClient.java:0)
at com/documentum/com/DfClientX.getLocalClient()Lcom/documentum/fc/client/IDfClient;(DfClientX.java:43)
at com/documentum/web/env/NotificationManager.contextInitialized(Ljavax/servlet/ServletContextEvent;)V(NotificationManager.java:101)
at weblogic/servlet/internal/EventsManager$FireContextListenerAction.run()Ljava/lang/Object;(EventsManager.java:458)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(Lweblogic/security/subject/AbstractSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(AuthenticatedSubject.java:321)
at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(Unknown Source)
at weblogic/servlet/internal/EventsManager.notifyContextCreatedEvent()V(EventsManager.java:168)
at weblogic/servlet/internal/WebAppServletContext.preloadResources()V(WebAppServletContext.java:1721)
^-- Holding lock: weblogic/servlet/internal/WebAppServletContext@0x0E953B38[thin lock]
at weblogic/servlet/internal/WebAppServletContext.start()V(WebAppServletContext.java:2890)
at weblogic/servlet/internal/WebAppModule.startContexts()V(WebAppModule.java:948)
at weblogic/servlet/internal/WebAppModule.start()V(WebAppModule.java:353)
at weblogic/application/internal/flow/ModuleStateDriver$3.next(Ljava/lang/Object;)V(ModuleStateDriver.java:204)
at weblogic/application/utils/StateMachineDriver.nextState(Lweblogic/application/utils/StateChange;[Ljava/lang/Object;)V(StateMachineDriver.java:26)
at weblogic/application/internal/flow/ModuleStateDriver.start([Lweblogic/application/Module;)V(ModuleStateDriver.java:60)
at weblogic/application/internal/flow/ScopedModuleDriver.start()V(ScopedModuleDriver.java:200)
at weblogic/application/internal/flow/ModuleListenerInvoker.start()V(ModuleListenerInvoker.java:117)
at weblogic/application/internal/flow/ModuleStateDriver$3.next(Ljava/lang/Object;)V(ModuleStateDriver.java:204)
at weblogic/application/utils/StateMachineDriver.nextState(Lweblogic/application/utils/StateChange;[Ljava/lang/Object;)V(StateMachineDriver.java:26)
at weblogic/application/internal/flow/ModuleStateDriver.start([Lweblogic/application/Module;)V(ModuleStateDriver.java:60)
at weblogic/application/internal/flow/StartModulesFlow.activate()V(StartModulesFlow.java:26)
at weblogic/application/internal/BaseDeployment$2.next(Ljava/lang/Object;)V(BaseDeployment.java:635)
at weblogic/application/utils/StateMachineDriver.nextState(Lweblogic/application/utils/StateChange;[Ljava/lang/Object;)V(StateMachineDriver.java:26)
at weblogic/application/internal/BaseDeployment.activate(Lweblogic/deploy/container/DeploymentContext;)V(BaseDeployment.java:212)
at weblogic/application/internal/DeploymentStateChecker.activate(Lweblogic/deploy/container/DeploymentContext;)V(DeploymentStateChecker.java:154)
at weblogic/deploy/internal/targetserver/AppContainerInvoker.activate(Lweblogic/deploy/container/DeploymentContext;)V(AppContainerInvoker.java:80)
at weblogic/deploy/internal/targetserver/BasicDeployment.activate(Lweblogic/deploy/container/DeploymentContext;)V(BasicDeployment.java:182)
at weblogic/deploy/internal/targetserver/BasicDeployment.activateFromServerLifecycle()V(BasicDeployment.java:359)
at weblogic/management/deploy/internal/DeploymentAdapter$1.doActivate(Ljava/lang/Object;)V(DeploymentAdapter.java:51)
at weblogic/management/deploy/internal/DeploymentAdapter.activate(Ljava/lang/Object;)V(DeploymentAdapter.java:196)
at weblogic/management/deploy/internal/AppTransition$2.transitionApp(Lweblogic/management/deploy/internal/DeploymentAdapter;Ljava/lang/Object;)V(AppTransition.java:30)
at weblogic/management/deploy/internal/ConfiguredDeployments.transitionApps(Lweblogic/management/deploy/internal/AppTransition;)V(ConfiguredDeployments.java:233)
at weblogic/management/deploy/internal/ConfiguredDeployments.activate()V(ConfiguredDeployments.java:169)
at weblogic/management/deploy/internal/ConfiguredDeployments.deploy()V(ConfiguredDeployments.java:123)
at weblogic/management/deploy/internal/DeploymentServerService.resume()V(DeploymentServerService.java:173)
at weblogic/management/deploy/internal/DeploymentServerService.start()V(DeploymentServerService.java:89)
at weblogic/t3/srvr/SubsystemRequest.run()V(SubsystemRequest.java:64)
at weblogic/work/ExecuteThread.execute(Ljava/lang/Runnable;)V(ExecuteThread.java:200)
at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:172)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace
Open lock chains
================
Chain 1:
"Main Thread" id=1 idx=0x2 tid=8140 waiting for java/lang/Class@0x00DB5430 held by:
"(Signal Handler)" id=2 idx=0x4 tid=1196 (waiting on notification)
Chain 2:
"Thread-1" id=10 idx=0x54 tid=1624 waiting for weblogic/servlet/internal/WebAppServletContext@0x0E953B38 held by:
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=13 idx=0x1e tid=732 (active)
===== END OF THREAD DUMP ===============
>
There is enough memory, and the CPU usaage does not get too high.
I have searched the forums and cant find what could be causing the blocking.
The startup hangs at the following line :
<Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
doing a Ctrl-C to stop the startup then produces the dump entries in the adminserver.log file :
There is no error or warning in the adminserver.log file.
What should I be checking to find out what is going on ? is this a known issue ?
Thanks.

Answer 1

Hi
1. Looking carefully at the following 3 lines in server statup logs, after 2:20:39 PM GST, server waited like for 16 mts like 2:36:43 and invoked the Force Shutdown. Possible reasons I gave below.
####<Feb 14, 2010 2:20:37 PM GST> <Info> <netuix> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <1266142837729> <BEA-423101> <[console] Initializing the NetUIx container>
####<Feb 14, 2010 2:20:39 PM GST> <Info> <netuix> <ePIMSEDMS2> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <1266142839433> <BEA-423120> <WEB-INF/client-classifications.xml file not found at uri [webapp]. Classifications functionality not enabled.>
####<Feb 14, 2010 2:36:43 PM GST> <Notice> <WebLogicServer> <ePIMSEDMS2> <AdminServer> <Thread-1> <<WLS Kernel>> <1266143803604> <BEA-000388> <JVM called WLS shutdown hook. The server will force shutdown now>
2. Since you mentioned this worked fine and nothing else is changed over a year, the possible cause may be like this. Looks like you have some Documentum stuff (or webapp or modules) loaded on this server. I am not much familiar with EMC Documentum stuff, except that we use that as external Content Management repository for Portal Applications. Looking at the full thread dump for thread 0, looks like documentum code is trying to publish something or interact with its documentum server and may be it is not getting the response back. So make sure if any external systems that this WLS interacts are all up and running.
HTH
Ravi Jegga