Thread waiting for monitor - appending to previous post

04:25:51,721 ERROR [ExecuteThread: '55' for queue: 'weblogic.kernel.Default'] com.documentum.web.common.Trace - SessionSynch@bc53ac[lockCount=2,lockOwner=Thread[ExecuteThread: '54' for queue: 'weblogic.kernel.Default',5,Thread Group for Queue: 'weblogic.kernel.Default']]: Timed out waiting in lock()
java.lang.IllegalStateException: SessionSynch@bc53ac[lockCount=2,lockOwner=Thread[ExecuteThread: '54' for queue: 'weblogic.kernel.Default',5,Thread Group for Queue: 'weblogic.kernel.Default']]: Timed out waiting in lock()
at com.documentum.web.common.SessionSync.lock0(SessionSync.java:181)
at com.documentum.web.common.SessionSync.lock(SessionSync.java:68)
at com.documentum.web.form.FormProcessor.lockSession(FormProcessor.java:2026)
at com.documentum.web.form.FormProcessor.invokeMethod(FormProcessor.java:1614)
at com.documentum.web.form.FormProcessor.fireOnExitEvent(FormProcessor.java:1463)
at com.documentum.web.form.FormProcessor.fireOnExitEvent(FormProcessor.java:1459)
at com.documentum.web.form.FormProcessor.fireOnExitEvent(FormProcessor.java:1438)
at com.documentum.web.form.JumpOperation.execute(JumpOperation.java:80)
at com.documentum.web.form.FormProcessor.openForm(FormProcessor.java:219)
at com.documentum.web.form.WebformTag.doStartTag(WebformTag.java:127)
at jsp_servlet._wdk.__historyreleased._jsp__tag0(__historyreleased.java:164)
at jsp_servlet._wdk.__historyreleased._jspService(__historyreleased.java:125)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:525)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:454)
at com.documentum.web.form.FormProcessor.dispatchURL(FormProcessor.java:2194)
at com.documentum.web.form.FormProcessor.doFreshInclude(FormProcessor.java:1082)
at com.documentum.web.form.FormProcessor.doFreshInclude(FormProcessor.java:1058)
at com.documentum.web.form.FormProcessor.redirect(FormProcessor.java:1010)
at com.documentum.web.form.FormProcessor.openForm(FormProcessor.java:257)
at com.documentum.web.form.WebformTag.doStartTag(WebformTag.java:127)
at jsp_servlet._webcomponent._library._locator.__locatorcontainer._jsp__tag2(__locatorcontainer.java:704)
at jsp_servlet._webcomponent._library._locator.__locatorcontainer._jsp__tag1(__locatorcontainer.java:671)
at jsp_servlet._webcomponent._library._locator.__locatorcontainer._jspService(__locatorcontainer.java:154)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.documentum.web.servlet.ResponseHeaderControlFilter.doFilter(ResponseHeaderControlFilter.java:317)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.documentum.web.servlet.CompressionFilter.doFilter(CompressionFilter.java:84)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.documentum.web.env.WDKController.processRequest(WDKController.java:95)
at com.documentum.web.env.WDKController.doFilter(WDKController.java:83)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Yes Jay, You are right. I have taken thread dump 10 samples and i could see that Thread State is not moving... I have increased the StuckThread Timeout to 800 from 600
Thread dump 1/10 "ExecuteThread: '32' for queue: 'weblogic.kernel.Default'" daemon prio=3 tid=0x01b28350 nid=0xa9 waiting for monitor entry [0x71e3e000..0x71e3fc70] at com.documentum.wp.app.WpRoleService.getRoleModel(WpRoleService.java:174) - waiting to lock <0x9c986ed0> (a java.lang.Object) at com.documentum.wp.app.WpRoleService.getUserRole(WpRoleService.java:92) at com.documentum.wp.session.LoginUtil.validateUserDomain(LoginUtil.java:58) at com.documentum.wp.session.WpAuthenticationService.login(WpAuthenticationService.java:70) at com.documentum.web.formext.session.Login.authenticate(Login.java:1041) at com.documentum.web.formext.session.Login.onLogin(Login.java:359) at sun.reflect.GeneratedMethodAccessor119.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at com.documentum.web.form.FormProcessor.invokeMethod(FormProcessor.java:1633) at com.documentum.web.form.FormProcessor.invokeMethod(FormProcessor.java:1487) at com.documentum.web.form.FormProcessor.fireActionEvent(FormProcessor.java:1303) at com.documentum.web.form.RecallOperation.execute(RecallOperation.java:101) at com.documentum.web.form.FormProcessor.processAction(FormProcessor.java:113) at com.documentum.web.form.FormAction.processAction(FormAction.java:107) at com.documentum.web.env.WDKController.doStartRequest(WDKController.java:191) at com.documentum.web.env.WDKController.processRequest(WDKController.java:92) at com.documentum.web.env.WDKController.doFilter(WDKController.java:83) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(Unknown Source) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366) at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Similar Messages

Jstack shows a thread waiting for monitor entry but it's already acquired

I have a heavy load website powered by tomcat-6.0.14 and jdk1.6.0_02-b05(amd64) on a 2 dual-core amd Opteron. When tomcat started, and I directed the http request(~200/s) abruptly to tomcat, it looks choked for over one minute, and if I took off the http requests, tomcat can recover in a few minutes.
Then I use jstack during the choke time to find what's the cause, where I found a strange situation. Almost all threads are blocking on getting a log4j lock(I think it's normal,since log4j use a coarse lock), but one thread which already got the lock are also blocked. Below is the stack trace.
"http-172.23.24.29-8011-exec-175" daemon prio=10 tid=0x00002aad37843400 nid=0x542d waiting for monitor entry [0x0000000052e55000..0x0000000052e55da0]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:201)
- locked <0x00002aaab39d5670> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.log(Category.java:853)
at org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:133)
at com.xxx.xxxx.TimingLogFilter.doFilter(TimingLogFilter.java:39)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:887)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:696)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2009)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
And all the other threads waiting for the log4j lock.
"http-172.23.24.29-8011-exec-176" daemon prio=10 tid=0x00002aad37773800 nid=0x542e waiting for monitor entry [0x0000000052f56000..0x0000000052f56b20]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:201)
- waiting to lock <0x00002aaab39d5670> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.log(Category.java:853)
at org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:133)
at com.xxx.xxxxx.TimingLogFilter.doFilter(TimingLogFilter.java:39)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:887)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:696)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2009)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
I'm really wondering why jstack shows the first thread is BLOCKED but no blocker.
And also, it's not a deadlock, since I made jstack log after sever seconds, and found this time another thread has the same problem.
"http-172.23.24.29-8011-exec-170" daemon prio=10 tid=0x00002aad3784d800 nid=0x5428 waiting for monitor entry [0x0000000052950000..0x0000000052950c20]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:201)
- locked <0x00002aaab39d5670> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.log(Category.java:853)
at org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:133)
at com.xxx.xxxx.TimingLogFilter.doFilter(TimingLogFilter.java:35)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:887)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:696)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2009)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
What's the cause? Is jstack accurate? Or, the problem is caused by hotspot compling since tomcat is just started? Or it's just because log4j's coarse lock? ( but after the starup, tomcat can handle the same traffic without any problem, and jstack shows no log4j block at all.)
Thanks in advance.
Jimcgnu

"http-172.23.24.29-8011-exec-175" daemon prio=10 tid=0x00002aad37843400 nid=0x542d waiting for monitor entry [0x0000000052e55000..0x0000000052e55da0]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:201)
- locked <0x00002aaab39d5670> (a org.apache.log4j.spi.RootLogger)
It seems like a Dead Lock.
The thread must have entered into a synchronized block/method1() and acquired a lock.
- locked <0x00002aaab39d5670> (a org.apache.log4j.spi.RootLogger)
While having lock the thread might be trying to acquire a lock to another synchronized block/method2()
waiting for monitor entry
java.lang.Thread.State: BLOCKED (on object monitor)
and someother thread might have already acquired a lock to that block/method2().
Can we check for Dead Lock??

Waiting for monitor enty messages in thread dump

Hi,
I've a thread dump of a container in a Oracle Application Server 9.0.4.1 (running 1.4.2_07) right before restart because off low memory and a lot of threads in this dump are in this state:
"AJPRequestHandler-ApplicationServerThread-1588" prio=1 tid=0x37634bf8 nid=0x1982 waiting for monitor entry [43f22000..43f2386c]
can anyone tell me what this means? Google didn't reveal anything useful.
Another thing I don't understand although it seems it doesn't do any harm:
"AJPRequestHandler-ApplicationServerThread-1630" prio=1 tid=0x09011a68 nid=0x1982 in Object.wait() [40276000..4027686c]
at java.lang.Object.wait(Native Method)
- waiting on <0x6624e628> (a java.util.HashMap)
at java.lang.Object.wait(Object.java:429)
at java.net.InetAddress.checkLookupTable(InetAddress.java:1109)
- locked <0x6624e628> (a java.util.HashMap)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1048)
at java.net.InetAddress.getLocalHost(InetAddress.java:1188)
at com.evermind.server.http.EvermindHttpSession.encodeSession(EvermindHttpSession.java:414)
at com.evermind.server.http.HttpApplication.createSession(HttpApplication.java:1599)
why is this thread waiting for a lock itself has?
Thanx for any help and hints in advance.
Regards,
Ingo

Hi,
I've a follow-up question, I have the following thread in several thread dumps:
"AJPRequestHandler-ApplicationServerThread-717" prio=1 tid=0x4046b988 nid=0x3dd3 waiting for monitor entry [3845e000..3845e86c]
     at java.net.Inet4Address.getAddress(Inet4Address.java:287)
     at com.evermind.server.http.StandardRequestLogger.appendIP(StandardRequestLogger.java:187)
     at com.evermind.server.http.StandardRequestLogger.log(StandardRequestLogger.java:45)
     - locked <0x663b2088> (a com.evermind.server.http.StandardRequestLogger)
     at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:906)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
     at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
     at java.lang.Thread.run(Thread.java:534)
What is the monitor this thread is waiting for? According to the source in Inet4Address at line 287 the method tries to create a new byte array... is there some system monitor which can't be seen in the thread dump?
This thread holds a lock (0x663b2088) 46 other threads are waiting for...
Thanx in advance for any hints.
Regards,
Ingo

JVM Stack trace shows waiting for monitor entry threads

Hi everybody,
I've got a problem with a java application wich connects to an Oracle 11g database and runs in a SJSAS 9.1, it freezes twice a day. The JVM trace shows threads in waiting for monitor entry state. This is an example of the most of threads:
"httpSSLWorkerThread-8080-127" daemon prio=1 tid=0x0bdb2e30 nid=0x4930 waiting for monitor entry [0x10a9f000..0x10aa0da0]
at com.sun.enterprise.resource.AbstractResourcePool.freeUnenlistedResource(AbstractResourcePool.java:1055)
- waiting to lock <0x373c07a0> (a com.sun.enterprise.resource.SJSASResourcePool)
at com.sun.enterprise.resource.AbstractResourcePool.resourceClosed(AbstractResourcePool.java:972)
at com.sun.enterprise.resource.PoolManagerImpl.putbackResourceToPool(PoolManagerImpl.java:461)
at com.sun.enterprise.resource.PoolManagerImpl.resourceClosed(PoolManagerImpl.java:401)
at com.sun.enterprise.resource.LocalTxConnectionEventListener.connectionClosed(LocalTxConnectionEventListener.java:71)
at com.sun.gjc.spi.ManagedConnection.connectionClosed(ManagedConnection.java:654)
at com.sun.gjc.spi.base.ConnectionHolder.close(ConnectionHolder.java:201)
at utilidades.db.BeanBaseDatos.desconectar(BeanBaseDatos.java:139)
at utilidades.db.BeanBaseDatos.ejecutarConsulta5(BeanBaseDatos.java:354)
at turnos.agenda.gestionAgenda.consultarProfesionalesReserva(gestionAgenda.java:3192)
at turnos.servletControlMfPersonas.procesar(servletControlMfPersonas.java:243)
at turnos.servletControlMfPersonas.doPost(servletControlMfPersonas.java:25)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:270)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212)
at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:361)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Thanks in advance.
Pablo

Hi,
I'm using Sun Java System Application Server 9.1 under Red Hat Linux.
Edited by: Paul-K on Jan 20, 2009 5:40 AM

Waiting for Monitor Lock on java/util/zip/ZipFile@6C816668/6C816674

Hi,
My server is creating the Thread dumps. After analysing, I found the following logs. Can any thing tell me that why is it stuck? How can I resolve this problem?
Thanks In Advance.
at java/util/zip/ZipFile.getEntry(ZipFile.java:287(Compiled Code)) at weblogic/utils/classloaders/ZipClassFinder.getSource(ZipClassFinder.java:32(Compiled Code)) at weblogic/utils/classloaders/JarClassFinder.getSource(JarClassFinder.java:45(Compiled Code)) at weblogic/utils/classloaders/MultiClassFinder.getSource(MultiClassFinder.java:64(Compiled Code)) at weblogic/utils/classloaders/MultiClassFinder.getSource(MultiClassFinder.java:64(Compiled Code)) at weblogic/utils/classloaders/MultiClassFinder.getSource(MultiClassFinder.java:64(Compiled Code)) at weblogic/utils/classloaders/MultiClassFinder.getSource(MultiClassFinder.java:64(Compiled Code)) at weblogic/utils/classloaders/MultiClassFinder.getSource(MultiClassFinder.java:64(Compiled Code)) at weblogic/utils/classloaders/CodeGenClassFinder.getSource(CodeGenClassFinder.java:33(Compiled Code)) at weblogic/utils/classloaders/GenericClassLoader.findResource(GenericClassLoader.java:209(Compiled Code)) at java/lang/ClassLoader.getResource(ClassLoader.java:422(Compiled Code)) at weblogic/utils/classloaders/GenericClassLoader.getResource(GenericClassLoader.java:169(Compiled Code)) at weblogic/utils/classloaders/ChangeAwareClassLoader.getResource(ChangeAwareClassLoader.java:112(Compiled Code)) at com/opensymphony/xwork2/util/ClassLoaderUtil.getResource(ClassLoaderUtil.java:83(Compiled Code)) at org/apache/struts2/views/freemarker/StrutsClassTemplateLoader.getURL(StrutsClassTemplateLoader.java:33(Compiled Code)) at freemarker/cache/URLTemplateLoader.findTemplateSource(URLTemplateLoader.java:75(Compiled Code)) at freemarker/cache/MultiTemplateLoader.findTemplateSource(MultiTemplateLoader.java:111(Compiled Code)) at freemarker/cache/TemplateCache.acquireTemplateSource(TemplateCache.java:600(Compiled Code)) at freemarker/cache/TemplateCache.findTemplateSource(TemplateCache.java:576(Compiled Code)) at freemarker/cache/TemplateCache.getTemplate(TemplateCache.java:293(Compiled Code)) at freemarker/cache/TemplateCache.getTemplate(TemplateCache.java:235(Compiled Code)) at freemarker/template/Configuration.getTemplate(Configuration.java:487(Compiled Code)) at freemarker/core/Environment.getTemplateForInclusion(Environment.java:1464(Compiled Code)) at freemarker/core/Include.accept(Include.java:157(Compiled Code)) at freemarker/core/Environment.visit(Environment.java:209(Compiled Code)) at freemarker/core/MixedContent.accept(MixedContent.java:92(Compiled Code)) at freemarker/core/Environment.visit(Environment.java:209(Compiled Code)) at freemarker/core/Environment.process(Environment.java:189(Compiled Code)) at freemarker/template/Template.process(Template.java:237(Compiled Code)) at org/apache/struts2/components/template/FreemarkerTemplateEngine.renderTemplate(FreemarkerTemplateEngine.java:148(Compiled Code)) at org/apache/struts2/components/UIBean.mergeTemplate(UIBean.java:530(Compiled Code)) at org/apache/struts2/components/UIBean.end(UIBean.java:484(Compiled Code)) at org/apache/struts2/views/jsp/ComponentTagSupport.doEndTag(ComponentTagSupport.java:43(Compiled Code)) at jsp_servlet/_web_45_inf/_decorators/__tpmsdecorator._jspService(__tpmsdecorator.java:475(Compiled Code)) at weblogic/servlet/jsp/JspBase.service(JspBase.java:34(Compiled Code)) at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:223(Compiled Code)) at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125(Compiled Code)) at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283(Compiled Code)) at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175(Compiled Code)) at weblogic/servlet/internal/RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:500(Compiled Code)) at weblogic/servlet/internal/RequestDispatcherImpl.include(RequestDispatcherImpl.java:431) at com/opensymphony/module/sitemesh/filter/PageFilter.writeDecorator(PageFilter.java:173) at com/opensymphony/module/sitemesh/filter/PageFilter.applyDecorator(PageFilter.java:158) at com/opensymphony/module/sitemesh/filter/PageFilter.doFilter(PageFilter.java:62) at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42(Compiled Code)) at org/acegisecurity/util/FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265(Compiled Code)) at org/acegisecurity/intercept/web/FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107) at org/acegisecurity/intercept/web/FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72) at org/acegisecurity/util/FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275(Compiled Code)) at org/acegisecurity/ui/ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110) at org/acegisecurity/util/FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275(Compiled Code)) at org/acegisecurity/wrapper/SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81) at org/acegisecurity/util/FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275(Compiled
Regards,
Developer.

Hi,
I've a follow-up question, I have the following thread in several thread dumps:
"AJPRequestHandler-ApplicationServerThread-717" prio=1 tid=0x4046b988 nid=0x3dd3 waiting for monitor entry [3845e000..3845e86c]
     at java.net.Inet4Address.getAddress(Inet4Address.java:287)
     at com.evermind.server.http.StandardRequestLogger.appendIP(StandardRequestLogger.java:187)
     at com.evermind.server.http.StandardRequestLogger.log(StandardRequestLogger.java:45)
     - locked <0x663b2088> (a com.evermind.server.http.StandardRequestLogger)
     at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:906)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
     at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
     at java.lang.Thread.run(Thread.java:534)
What is the monitor this thread is waiting for? According to the source in Inet4Address at line 287 the method tries to create a new byte array... is there some system monitor which can't be seen in the thread dump?
This thread holds a lock (0x663b2088) 46 other threads are waiting for...
Thanx in advance for any hints.
Regards,
Ingo

App Server Freeze - All threads waiting for replication data

          We are running Weblogic 6.1 SP2, on solaris 8 with jdk1.3 When running load at our
          application - full GC is taking between 15 -20 seconds. At a certain point after
          a full GC on one of the servers in the cluster we see session replication errors
          in the logs. The application eventually freezes - the execute queue is huge on each
          of the JVMS and when we do a thread dump we see that all threads are waiting for
          replication data. All threads have the same stack trace as follows:
          "ExecuteThread: '1' for queue: 'default'" daemon prio=5 tid=0x2c58b0 nid=0xe waiting
          on monitor [0xbf980000..0xbf9819d8]
               at java.lang.Object.wait(Native Method)
               at weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:72)
               at weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:97)
               at weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:80)
               at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:135)
               at weblogic.rmi.internal.ProxyStub.invoke(ProxyStub.java:35)
               at $Proxy74.lookupROIDS(Unknown Source)
               at weblogic.servlet.internal.HttpServer.lookupROIDS(HttpServer.java:761)
               at weblogic.servlet.internal.session.ReplicatedSessionContext.getSessionInternal(ReplicatedSessionContext.java:225)
               at weblogic.servlet.internal.ServletRequestImpl.getValidSession(ServletRequestImpl.java:2130)
               at weblogic.servlet.internal.ServletRequestImpl.getSession(ServletRequestImpl.java:1921)
               at weblogic.servlet.security.internal.ServletSecurityManager.checkAccess(ServletSecurityManager.java:107)
               at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:2467)
               at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2204)
               at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
               at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)


          Yes this is reproducable. We are now working with BEA Support under Case : 321199.
          Thanks
          Anna
          Kumar Allamraju <[email protected]> wrote:
          >You should work with [email protected]
          >Can this problem be reproduced with an isolated test case?
          >If yes, it will be really helpful for them to work with Engg.
          >
          >--
          >Kumar
          >
          >anna bancroft wrote:
          >
          >> We are running Weblogic 6.1 SP2, on solaris 8 with jdk1.3 When running
          >load at our
          >> application - full GC is taking between 15 -20 seconds. At a certain
          >point after
          >> a full GC on one of the servers in the cluster we see session replication
          >errors
          >> in the logs. The application eventually freezes - the execute queue is
          >huge on each
          >> of the JVMS and when we do a thread dump we see that all threads are waiting
          >for
          >> replication data. All threads have the same stack trace as follows:
          >> "ExecuteThread: '1' for queue: 'default'" daemon prio=5 tid=0x2c58b0 nid=0xe
          >waiting
          >> on monitor [0xbf980000..0xbf9819d8]
          >>      at java.lang.Object.wait(Native Method)
          >>      at weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:72)
          >>      at weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:97)
          >>      at weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:80)
          >>      at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:135)
          >>      at weblogic.rmi.internal.ProxyStub.invoke(ProxyStub.java:35)
          >>      at $Proxy74.lookupROIDS(Unknown Source)
          >>      at weblogic.servlet.internal.HttpServer.lookupROIDS(HttpServer.java:761)
          >>      at weblogic.servlet.internal.session.ReplicatedSessionContext.getSessionInternal(ReplicatedSessionContext.java:225)
          >>      at weblogic.servlet.internal.ServletRequestImpl.getValidSession(ServletRequestImpl.java:2130)
          >>      at weblogic.servlet.internal.ServletRequestImpl.getSession(ServletRequestImpl.java:1921)
          >>      at weblogic.servlet.security.internal.ServletSecurityManager.checkAccess(ServletSecurityManager.java:107)
          >>      at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:2467)
          >>      at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2204)
          >>      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
          >>      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
          >>
          >>
          >

LVM waits for seven minutes (see latest post)

Once upon a time my computer powered up in ~40 seconds. Alas those days are gone! Nowadays there is a seven minute (approx.) pause between GRUB and any output on the screen.
There are no errors in any log or dmesg that I can find. After GRUB runs there appears to be IO activity and a blinking cursor.
I *think* this started happening shortly after an upgrade.
# uname -a
Linux mediapc 3.1.9-1-ARCH #1 SMP PREEMPT Fri Jan 13 12:43:42 CET 2012 x86_64 AMD E-350 Processor AuthenticAMD GNU/Linux
Any ideas?
Last edited by fumbles (2012-01-20 13:57:23)

thisoldman wrote:
People will need to know if you are booting to console or X. Log-in manager?
Have you tried booting with the fallback kernel or another OS to compare?
Have you any ignored pacnew or pacsave files? Any alerts in the pacman log that you might have missed?
Have you checked for filesystem errors?
Booting to X, Y or Z make no difference as the seven minute pause is during the load of the kernel and once loaded everything works fine.
I have tried the fallback kernel and just booting to single user, both have a seven minute pause. I don't have any other OS to boot.
Yes and most likely, and they would be gone now due to logrotate.
fsck returns ok.
vacant wrote:
Is the disk access light on during the delay? I've had this when the BIOS thinks the disk is dirty, like after a power cut.
If so I'd check your shutdown procedure, try going to "init 1" first, do a "sync" then watch for messages after typing "halt" (or shutdown, reboot etc).
Why yes it is! I'll give it a go
Thanks for your help

Please thank everyone for responding to my previous posts

I finally got another email. I was removed from the discussion and had several suggested readings regarding use of forum. I had inadvertently done something that I did not know was illegal per the advice of a friend who I thought was more knowledgeable than she was, and now I guess I've "jailbroken" my phone. Now no one will help me. I'll have to resort to looking for answers from all the wackos on the internet, some of which may not have my best interest in mind. I'll have to take my chances. Please thank everyone for responding to my inquiry regarding FelipeV. I guess I'm now banned from discussions.

Thank you again. Last night I reopened iTunes and it said it had to run in compatibility mode. I was caught of guard by that and tried to figure out what was going on because I did not want to chance messing up my iPhone because it wasn't perfect. I tried finding articles to fix it and finally decided to delete the app using the instructions from support.apple.com. While attempting to delete the first program in the specific instructed order, I received error message while removing iTunes. Error code 2324 and then Error writing to Program Files (86) iTunes IT Detector.OCX. I spent hours doing that as it would appear to uninstall and then not do it. I can't remember what I used Fix It for but it was stuck in a loop for hours and finally I stopped it. I ran a scan search and found OCX file and changed security permission and retried what I was doing and all worked. AT that point I was really tired and I don't remember how I finally did it, but once the first application (iTunes) was deleted I had no problem uninstalling the other four related applications. After restarting, I looked in Program Files (86) to see if any files were left behind and didn't see any. I did not want to mess with the registry for fear I'd totally mess things up. My computer did hang up on starting once and it went into repair mode and it fixed itself. I did a reboot, pushing my luck, to make sure it would start okay again, and then I did a malware scan with both Windows Malicious Tool Removal update app from Windows and also the free version of Malware Byte. Both came back clean. I then went to bed at about 6:30 am. I plan on reinstalling iTunes after I make some coffee and see if it does a clean install. Its not like I don't maintain my computer. I did a factory reset and everything was working great. Hadn't even junked up my computer with all my document files yet. They're still on flash drive. I will run scan disc while having coffee. I've only used 71.5 of 451GB so it shouldn't take long. Thanks for your patience.

States in the thread dump "Waiting on monitor"

that is normal, it is all the execute threads waiting for requests
Rob Baldassano wrote:
Hi.
I am trying to do some troubleshooting of the weblogic server,
Currently we are using WebLogic 6.1 SP 2.
In the thread dump we are seeing a lot of "waiting on monitor" messages for the
individual threads.
The application that we are running uses native threads to run the processes that
it needs to.
My question is this.
Is this "Waiting for monitor" message normal?
What could cause these messages?
is there a way to get around them, besides recoding the application?

Waiting for many threads

Hey guys,
I know how to make one thread wait for another named thread to complete, but how do u make a thread wait for many threads to complete with the added problem of not knowing the names of the threads your waiting for? Looked and been trying for ages but can't get anything to work.
Thanx
Lisa

No i saw it, pehaps i should rephrase with a question, how would you go about giving all these threads a reference? If it ain't obvious already am not great with java and if someone could tell me how to give the threads spawned references it would be great.
As ive explained the code is really long so am not going to waist peoples time by posting it. Here is basically how i am spawning the threads "willy-nilly" style.
     while ((p < searchTerms.size())&&(p < 30)){
         term = (searchTerms.elementAt(p)).toString();
         NetChecker nc = new NetChecker(term,excludedAdds,refWeb);
         searchClasses.addElement(nc);
         new Thread(nc).start();
         p++;
     } the classes all return web addresses in a vector, thats why i need all the threads to complete so i can collect all the results
Thanx
Lisa

Infinite wait for connection using JDBC thin driver.

Hi,
I am using an OracleDataSource and an OracleConnectionCacheManager to manage my connection pool from my stand alone Java application. I am using ojdbc14.jar for my Oracle drivers. The database is Oracle 10g RAC. Java version is 1.5. I am using thin client.
The problem I have experienced is that upon calling the getConnection() method on the OracleDataSource object, my application waits infinitely for a connection. As my application is not checking how long it takes to retrieve a connection, the application effectively hangs.
Has anyone else witnessed this behaviour?? I have observed this behaviour in two seperate stand alone Java applications that are both talking to Oracle RAC databases.
When I create the OracleDataSource, I am considering calling the setLoginTimeout() method to specify a timeout when trying to connect to the database. Does anyone know if this will throw an SQLException if the infinite wait occurs again??
Thanks, Kenny.

OK, I have the JVM stack trace at last!!! Here it is. This actually uses the following code every single time a database connection is required.
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
conn = (OracleConnection)DriverManager.getConnection(url, username, password);Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_15-b04 mixed mode):
"pool-2-thread-3" prio=10 tid=0x0000000100fd2500 nid=0x68 waiting for monitor entry [0xffffffff1cdfe000..0xffffffff1cdff628]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-2-thread-2" prio=10 tid=0x0000000100ecfb00 nid=0x67 waiting for monitor entry [0xffffffff1d0fe000..0xffffffff1d0ff7a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-21" prio=10 tid=0x0000000100fa9180 nid=0x66 waiting for monitor entry [0xffffffff1e4fe000..0xffffffff1e4ff728]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-20" prio=10 tid=0x0000000100f832b0 nid=0x65 waiting for monitor entry [0xffffffff1e6fe000..0xffffffff1e6ff8a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-19" prio=10 tid=0x0000000100f82240 nid=0x64 runnable [0xffffffff1e8fe000..0xffffffff1e8ff828]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:129)
     at oracle.net.ns.Packet.receive(Unknown Source)
     at oracle.net.ns.NSProtocol.connect(Unknown Source)
     at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:707)
     at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:218)
     at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:343)
     at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:147)
     at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:31)
     at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:545)
     at java.sql.DriverManager.getConnection(DriverManager.java:525)
     - locked <0xffffffff2045e108> (a java.lang.Class)
     at java.sql.DriverManager.getConnection(DriverManager.java:171)
     - locked <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-18" prio=10 tid=0x0000000100f905a0 nid=0x63 waiting for monitor entry [0xffffffff1ebfe000..0xffffffff1ebff9a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-17" prio=10 tid=0x0000000101a531b0 nid=0x62 waiting for monitor entry [0xffffffff1edfe000..0xffffffff1edff928]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-16" prio=10 tid=0x000000010171ced0 nid=0x61 waiting for monitor entry [0xffffffff1f0fe000..0xffffffff1f0ff6a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-15" prio=10 tid=0x0000000100ea6a70 nid=0x60 waiting for monitor entry [0xffffffff1f2fe000..0xffffffff1f2ff628]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-14" prio=10 tid=0x0000000101a520b0 nid=0x5f waiting for monitor entry [0xffffffff1f4fe000..0xffffffff1f4ff7a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-13" prio=10 tid=0x0000000100e7a870 nid=0x5e waiting for monitor entry [0xffffffff1f6fe000..0xffffffff1f6ff728]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-12" prio=10 tid=0x0000000100ea7430 nid=0x5d waiting for monitor entry [0xffffffff1f8fe000..0xffffffff1f8ff8a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-11" prio=10 tid=0x0000000101262330 nid=0x5c waiting for monitor entry [0xffffffff1fafe000..0xffffffff1faff828]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-10" prio=10 tid=0x0000000100b66160 nid=0x5b waiting for monitor entry [0xffffffff1fcfe000..0xffffffff1fcff9a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-9" prio=10 tid=0x0000000100b47ff0 nid=0x5a waiting for monitor entry [0xffffffff1fefe000..0xffffffff1feff928]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-8" prio=10 tid=0x0000000101a7ac70 nid=0x59 waiting for monitor entry [0xffffffff65afe000..0xffffffff65aff6a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-7" prio=10 tid=0x0000000101212460 nid=0x58 waiting for monitor entry [0xffffffff65cfe000..0xffffffff65cff628]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-6" prio=10 tid=0x0000000100f8f6f0 nid=0x57 waiting for monitor entry [0xffffffff65efe000..0xffffffff65eff7a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-5" prio=10 tid=0x0000000100c7c420 nid=0x56 waiting for monitor entry [0xffffffff660fe000..0xffffffff660ff728]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-2-thread-1" prio=10 tid=0x0000000101a41370 nid=0x55 waiting for monitor entry [0xffffffff662fe000..0xffffffff662ff8a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-4" prio=10 tid=0x0000000101a28ef0 nid=0x54 waiting for monitor entry [0xffffffff664fe000..0xffffffff664ff828]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-3" prio=10 tid=0x00000001009a1850 nid=0x53 waiting for monitor entry [0xffffffff666fe000..0xffffffff666ff9a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-2" prio=10 tid=0x0000000100f8e980 nid=0x52 waiting for monitor entry [0xffffffff668fe000..0xffffffff668ff928]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"pool-1-thread-1" prio=10 tid=0x0000000100bd86f0 nid=0x51 waiting for monitor entry [0xffffffff67afe000..0xffffffff67aff6a8]
     at java.sql.DriverManager.registerDriver(DriverManager.java:259)
     - waiting to lock <0xffffffff2045e108> (a java.lang.Class)
     at com.jpm.mra.common.DBUtil.getDBConnection(Unknown Source)
     at com.jpm.mra.bulking.BaseBulker.openConnection(Unknown Source)
     at com.jpm.mra.bulking.GenericBulker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
     at java.lang.Thread.run(Thread.java:595)
"Thread-5" prio=10 tid=0x0000000100b66b90 nid=0x50 waiting on condition [0xffffffff67cff000..0xffffffff67cff628]
     at java.lang.Thread.sleep(Native Method)
     at com.jpm.mra.bulking.IntervalBulker.run(Unknown Source)
     at java.lang.Thread.run(Thread.java:595)
"Thread-3" prio=10 tid=0x0000000100f36860 nid=0x4f waiting on condition [0xffffffff67eff000..0xffffffff67eff7a8]
     at java.lang.Thread.sleep(Native Method)
     at com.jpm.mra.bulking.ClockBulker.run(Unknown Source)
     at java.lang.Thread.run(Thread.java:595)
"Thread-1" prio=10 tid=0x0000000100a4b300 nid=0x4e waiting on condition [0xffffffff680ff000..0xffffffff680ff728]
     at java.lang.Thread.sleep(Native Method)
     at com.jpm.mra.bulking.DepthBulker.run(Unknown Source)
     at java.lang.Thread.run(Thread.java:595)
"Thread-4" prio=10 tid=0x0000000100a4c380 nid=0x4d waiting on condition [0xffffffff682ff000..0xffffffff682ff8a8]
     at java.lang.Thread.sleep(Native Method)
     at com.jpm.mra.bulking.IntervalBulkerStaticDataPoller.run(Unknown Source)
     at java.lang.Thread.run(Thread.java:595)
"Thread-2" prio=10 tid=0x0000000100bcf260 nid=0x4c waiting on condition [0xffffffff684ff000..0xffffffff684ff828]
     at java.lang.Thread.sleep(Native Method)
     at com.jpm.mra.bulking.ClockBulkerStaticDataPoller.run(Unknown Source)
     at java.lang.Thread.run(Thread.java:595)
"Thread-0" prio=10 tid=0x0000000100a1b2d0 nid=0x4b waiting on condition [0xffffffff689ff000..0xffffffff689ff9a8]
     at java.lang.Thread.sleep(Native Method)
     at com.jpm.mra.bulking.MRABulking.run(Unknown Source)
     at java.lang.Thread.run(Thread.java:595)
"Low Memory Detector" daemon prio=10 tid=0x00000001006db880 nid=0x49 runnable [0x0000000000000000..0x0000000000000000]
"CompilerThread1" daemon prio=10 tid=0x00000001006da9e0 nid=0x48 waiting on condition [0x0000000000000000..0xffffffff6b6fdcc8]
"CompilerThread0" daemon prio=10 tid=0x00000001006d72a0 nid=0x47 waiting on condition [0x0000000000000000..0xffffffff6b8fde58]
"AdapterThread" daemon prio=10 tid=0x00000001006d6370 nid=0x46 waiting on condition [0x0000000000000000..0x0000000000000000]
"Signal Dispatcher" daemon prio=10 tid=0x00000001006d5440 nid=0x45 waiting on condition [0x0000000000000000..0x0000000000000000]
"Finalizer" daemon prio=10 tid=0x00000001006bf6b0 nid=0x44 in Object.wait() [0xffffffff6beff000..0xffffffff6beff828]
     at java.lang.Object.wait(Native Method)
     - waiting on <0xffffffff2544e010> (a java.lang.ref.ReferenceQueue$Lock)
     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
     - locked <0xffffffff2544e010> (a java.lang.ref.ReferenceQueue$Lock)
     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
     at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00000001006bd500 nid=0x43 in Object.wait() [0xffffffff6c0ff000..0xffffffff6c0ff9a8]
     at java.lang.Object.wait(Native Method)
     - waiting on <0xffffffff2544e000> (a java.lang.ref.Reference$Lock)
     at java.lang.Object.wait(Object.java:474)
     at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
     - locked <0xffffffff2544e000> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x000000010011ed20 nid=0x1 in Object.wait() [0xffffffff7fff5000..0xffffffff7fff56e0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0xffffffff25456218> (a java.lang.Thread)
     at java.lang.Thread.join(Thread.java:1095)
     - locked <0xffffffff25456218> (a java.lang.Thread)
     at java.lang.Thread.join(Thread.java:1148)
     at com.jpm.mra.bulking.MRABulking.main(Unknown Source)
"VM Thread" prio=10 tid=0x00000001006b9790 nid=0x42 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00000001001d47d0 nid=0x2 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00000001001d5270 nid=0x3 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00000001001d5d70 nid=0x4 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00000001001d6870 nid=0x5 runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x00000001001d7370 nid=0x6 runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x00000001001d7e70 nid=0x7 runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x00000001001d8970 nid=0x8 runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x00000001001d9470 nid=0x9 runnable
"GC task thread#8 (ParallelGC)" prio=10 tid=0x00000001001d9f70 nid=0xa runnable
"GC task thread#9 (ParallelGC)" prio=10 tid=0x00000001001daa70 nid=0xb runnable
"GC task thread#10 (ParallelGC)" prio=10 tid=0x00000001001db570 nid=0xc runnable
"GC task thread#11 (ParallelGC)" prio=10 tid=0x00000001001dc070 nid=0xd runnable
"GC task thread#12 (ParallelGC)" prio=10 tid=0x00000001001dcb70 nid=0xe runnable
"GC task thread#13 (ParallelGC)" prio=10 tid=0x00000001001dd670 nid=0xf runnable
"GC task thread#14 (ParallelGC)" prio=10 tid=0x00000001001de170 nid=0x10 runnable
"GC task thread#15 (ParallelGC)" prio=10 tid=0x00000001001dec70 nid=0x11 runnable
"GC task thread#16 (ParallelGC)" prio=10 tid=0x00000001001df7b0 nid=0x12 runnable
"GC task thread#17 (ParallelGC)" prio=10 tid=0x00000001001e02b0 nid=0x13 runnable
"GC task thread#18 (ParallelGC)" prio=10 tid=0x00000001001e0db0 nid=0x14 runnable
"GC task thread#19 (ParallelGC)" prio=10 tid=0x00000001001e18b0 nid=0x15 runnable
"GC task thread#20 (ParallelGC)" prio=10 tid=0x00000001001e23b0 nid=0x16 runnable
"GC task thread#21 (ParallelGC)" prio=10 tid=0x00000001001e2eb0 nid=0x17 runnable
"GC task thread#22 (ParallelGC)" prio=10 tid=0x00000001001e39b0 nid=0x18 runnable
"GC task thread#23 (ParallelGC)" prio=10 tid=0x00000001001e44b0 nid=0x19 runnable
"GC task thread#24 (ParallelGC)" prio=10 tid=0x00000001001e4fb0 nid=0x1a runnable
"GC task thread#25 (ParallelGC)" prio=10 tid=0x00000001001e5ab0 nid=0x1b runnable
"GC task thread#26 (ParallelGC)" prio=10 tid=0x00000001001e65b0 nid=0x1c runnable
"GC task thread#27 (ParallelGC)" prio=10 tid=0x00000001001e70b0 nid=0x1d runnable
"GC task thread#28 (ParallelGC)" prio=10 tid=0x00000001001e7bb0 nid=0x1e runnable
"GC task thread#29 (ParallelGC)" prio=10 tid=0x00000001001e86b0 nid=0x1f runnable
"GC task thread#30 (ParallelGC)" prio=10 tid=0x00000001001e91b0 nid=0x20 runnable
"GC task thread#31 (ParallelGC)" prio=10 tid=0x00000001001e9cb0 nid=0x21 runnable
"GC task thread#32 (ParallelGC)" prio=10 tid=0x00000001001ea7b0 nid=0x22 runnable
"GC task thread#33 (ParallelGC)" prio=10 tid=0x00000001001eb2b0 nid=0x23 runnable
"GC task thread#34 (ParallelGC)" prio=10 tid=0x00000001001ebdb0 nid=0x24 runnable
"GC task thread#35 (ParallelGC)" prio=10 tid=0x00000001001ec8b0 nid=0x25 runnable
"GC task thread#36 (ParallelGC)" prio=10 tid=0x00000001001ed3b0 nid=0x26 runnable
"GC task thread#37 (ParallelGC)" prio=10 tid=0x00000001001edeb0 nid=0x27 runnable
"GC task thread#38 (ParallelGC)" prio=10 tid=0x00000001001f09c0 nid=0x28 runnable
"GC task thread#39 (ParallelGC)" prio=10 tid=0x00000001001f14c0 nid=0x29 runnable
"GC task thread#40 (ParallelGC)" prio=10 tid=0x00000001001f1fc0 nid=0x2a runnable
"GC task thread#41 (ParallelGC)" prio=10 tid=0x00000001001f2ac0 nid=0x2b runnable
"GC task thread#42 (ParallelGC)" prio=10 tid=0x00000001001f35c0 nid=0x2c runnable
"GC task thread#43 (ParallelGC)" prio=10 tid=0x00000001001f40c0 nid=0x2d runnable
"GC task thread#44 (ParallelGC)" prio=10 tid=0x00000001001f4bc0 nid=0x2e runnable
"GC task thread#45 (ParallelGC)" prio=10 tid=0x00000001001f56c0 nid=0x2f runnable
"GC task thread#46 (ParallelGC)" prio=10 tid=0x00000001001f61c0 nid=0x30 runnable
"GC task thread#47 (ParallelGC)" prio=10 tid=0x00000001001f6cc0 nid=0x31 runnable
"GC task thread#48 (ParallelGC)" prio=10 tid=0x00000001001f7810 nid=0x32 runnable
"GC task thread#49 (ParallelGC)" prio=10 tid=0x00000001001f8310 nid=0x33 runnable
"GC task thread#50 (ParallelGC)" prio=10 tid=0x00000001001f8e10 nid=0x34 runnable
"GC task thread#51 (ParallelGC)" prio=10 tid=0x00000001001f9910 nid=0x35 runnable
"GC task thread#52 (ParallelGC)" prio=10 tid=0x00000001001fa410 nid=0x36 runnable
"GC task thread#53 (ParallelGC)" prio=10 tid=0x00000001001faf10 nid=0x37 runnable
"GC task thread#54 (ParallelGC)" prio=10 tid=0x00000001001fba10 nid=0x38 runnable
"GC task thread#55 (ParallelGC)" prio=10 tid=0x00000001001fc510 nid=0x39 runnable
"GC task thread#56 (ParallelGC)" prio=10 tid=0x00000001001fd010 nid=0x3a runnable
"GC task thread#57 (ParallelGC)" prio=10 tid=0x00000001001fdb10 nid=0x3b runnable
"GC task thread#58 (ParallelGC)" prio=10 tid=0x00000001001fe610 nid=0x3c runnable
"GC task thread#59 (ParallelGC)" prio=10 tid=0x00000001001ff110 nid=0x3d runnable
"GC task thread#60 (ParallelGC)" prio=10 tid=0x00000001001ffc10 nid=0x3e runnable
"GC task thread#61 (ParallelGC)" prio=10 tid=0x0000000100200710 nid=0x3f runnable
"GC task thread#62 (ParallelGC)" prio=10 tid=0x0000000100201210 nid=0x40 runnable
"GC task thread#63 (ParallelGC)" prio=10 tid=0x0000000100201d10 nid=0x41 runnable
"VM Periodic Task Thread" prio=10 tid=0x00000001001c93f0 nid=0x4a waiting on condition
Edited by: krodman on 08-Jan-2010 04:03

Stopped threads waiting on DriverManager.getConnection()?

Solaris 5.8, Sun JDK 1.3.1-b24
RMI based application with the server piece running as a Java app on Solaris. After a few hours of use, with 100-150 users, the server app appears to hang from the client. If I do a kill -3 on the server process, I see a whole buttload of threads (over 100) stuck on DriverManager.getConnection() method (see below). The trace doesn't show anything deeper than that, but it is Oracle thin client being used. Our DBAs don't see any connections being attempted in the listeners log, and in fact, the number of the connections I see to the DB box via netstat jives exactly with the number of sessions I see on the Oracle side (in V$SESSION). While this is going on, I can connect to the DB box from the app server box all day long using a small class I wrote that just does a connection open and close.
This app was working fairly flawlessly until about a week ago. Nothing in terms of drivers or JVM version has changed, but there has been slightly more user load lately. It seems that every new user request that comes in after a certain point ends up with a thread blocked like this. What is the thread stopped in getConnection() indicative of? Has anyone seen anything like this?
Much appreciated,
Mike
Tons of these show in the thread dump (kill -3):
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:"RMI TCP Connection(11307)-10.160.96.220" daemon prio=5 tid=0x4946d0 nid=0x2e7b waiting for monitor entry [0xd4480000..0xd44819e0]
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:     at java.sql.DriverManager.getConnection(DriverManager.java:193)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:     at sched_MT.rmiSched_MTImpl.getLocation(rmiSched_MTImpl.java:556)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:     at java.lang.reflect.Method.invoke(Native Method)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:     at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:241)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:     at sun.rmi.transport.Transport$1.run(Transport.java:152)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:     at java.security.AccessController.doPrivileged(Native Method)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:     at sun.rmi.transport.Transport.serviceCall(Transport.java:148)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:     at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:465)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:     at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:706)

Yes - it is roughly 1 user : 1 DB connection. No pooling. Rogue app that we inherited. Ack. While it's blocking users, I can connect to the DBs fine (there are 2) from another app on the same box using the same DB logins and the same DBs, so my impression is that it's not DB related, but at this point, I am not completely sure about that. Our DBA says TNS is not getting any connect attempts during this time, so it seems like the driver is not even attempting.

Thread waiting to lock on an object that it has already locked

I have a thread in my thread dump where it is "waiting to lock" on an object, but it already has acquired the lock. How can this be possible?
"TP-Processor778" daemon prio=1 tid=0x230dde38 nid=0x1168 waiting for monitor entry [0x0f184000..0x0f185fa0]
at sun.net.www.protocol.file.Handler.openConnection(Handler.java:63)
- waiting to lock <0x40fbf128> (a sun.net.www.protocol.file.Handler)
at sun.net.www.protocol.file.Handler.openConnection(Handler.java:55)
- locked <0x40fbf128> (a sun.net.www.protocol.file.Handler)
at java.net.URL.openConnection(URL.java:943)
at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)

If object.wait() is called, an existing lock is released so another thead can lock the object.
If you object.notify() but hold the lock in another thread, the first thread will not resume until it has required the lock.

Waiting on Monitor

Hi,
I have so many threads waiting on monitor while I analyze the thread dump.
Is that mean anything (number of the threads waiting on monitor is high)
Thanks.
--RJ

Hi,
I know I am quite a bit late for your question, but maybe it will be of any help to others if not to you (any more...).
Multiple threads waiting on monitors do not necessarily have to indicate a problem. Problems arise e.g. if one thread holds monitor X and tries to acquire monitor Y, which in turn is held by another thread trying to acquire monitor X. This would be a classical deadlock as none of both threads will ever get to acquire the monitor it needs to do its work.
Threads may also wait on monitors simply waiting to be notified about pending work (e.g. incoming IO data), which is no problem at all, but intended.
Bye.

Memory usage in Thread wait()

Hi all,
I have a question.
What is the memory usage (in terms of RAM, not of virtual memory) if 1000 Threads wait for a long time on a common object suppose for more than one day?
Will be they moved to disk memory and then in RAM when a notify() occurs or what else? Thanks! Bye

Ok,
I understood your answers but I see the scenario is not clear.
Suppose a program has to launch and high number of threads. Some of these threads will wait on a monitor for a large period of time (in terms of days...) and others will have no wait at all.
Eg. Think as half the threads that run on the JVM blocks on a monitor and the other half not. Suppose we create 1000 thread every 5 second.
So we have this kind of evolution in one minute
12 * 1000 Threads are created
12 * 500 Thread will wait on a monitor (for a very long time)
What the JVM does with this thread? Will the JVM move this threads for RAM to disk or will it keep them in RAM?
In other words, what is the runtime memory usage of the JVM when a thread waits for a very long time? Will it temporary free the memory of the thread, saving it in the disk or will it keep it in RAM?
I hope I've been clear now! Bye

Thread waiting for monitor - appending to previous post

Similar Messages

Maybe you are looking for