Is-it normal to face situation where the "Reference Handler" is blocked?

Question

Hi all,
I would like to know if it can be a normal behavior to have the "Reference Handler" thread from the class java.lang.ref.Reference in BLOCKED state in my thread dump (generated from visualvm) as you can see below.
"http-0.0.0.0-8080-56" - Thread t@578
   java.lang.Thread.State: RUNNABLE
at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167)
at org.apache.lucene.index.SegmentMergeInfo.next(SegmentMergeInfo.java:66)
at org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.next(MultiSegmentReader.java:494)
at org.apache.lucene.index.SegmentMergeInfo.next(SegmentMergeInfo.java:65)
at org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.next(MultiSegmentReader.java:494)
at org.apache.lucene.search.FilteredTermEnum.next(FilteredTermEnum.java:67)
at org.apache.lucene.search.FilteredTermEnum.setEnum(FilteredTermEnum.java:49)
at org.apache.lucene.search.FuzzyTermEnum.<init>(FuzzyTermEnum.java:127)
at org.apache.lucene.search.FuzzyQuery.getEnum(FuzzyQuery.java:100)
at org.apache.lucene.search.FuzzyQuery.rewrite(FuzzyQuery.java:104)
at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:383)
at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:368)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.exoplatform.services.security.web.SetCurrentIdentityFilter.doFilter(SetCurrentIdentityFilter.java:90)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.exoplatform.portal.application.localization.LocalizationFilter.doFilter(LocalizationFilter.java:179)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.exoplatform.web.login.ClusteredSSOFilter.doFilter(ClusteredSSOFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.exoplatform.container.web.PortalContainerFilter.doFilter(PortalContainerFilter.java:69)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:402)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)
at java.lang.Thread.run(Thread.java:662)
   JNI locked monitors:
- locked <5258ed3e> (a java.lang.ref.Reference$Lock)
   Locked ownable synchronizers:
- None
"Reference Handler" - Thread t@2
   java.lang.Thread.State: BLOCKED
at java.lang.Object.wait(Native Method)
- waiting on <5258ed3e> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
   Locked ownable synchronizers:
- NoneAs you can see in my thread dump the object "java.lang.ref.Reference$Lock" is locked by a common thread which has for consequences to annoy a lot the GC and to freeze the whole application. Regarding the code (http://kickjava.com/src/java/lang/ref/Reference.java.htm) since the class java.lang.ref.Reference$Lock is private and the field "lock" is also (static) private, I was wondering if it is normal that this field can be locked by a common thread and if so can you tell me why and how (by reflection I assume like my example below)? Can it be a JVM Bug?
This issue makes me realized that any Java application without a Security Manager installed can be easily frozen thanks to this simple code:
      // The thread that will steal the lock to the "Reference Handler" to block the GC and then the whole application
      new Thread("Reference$Lock Holder Thread")
         public void run()
            try
               // Get the lock field by reflection
               Field fieldLock = Reference.class.getDeclaredField("lock");
               // Make it accessible
               fieldLock.setAccessible(true);
               // Get the static lock object
               Object o = fieldLock.get(null);
               Object mutex = new Object();
               // Lock the lock object
               synchronized (o)
                  // Wait forever
                  synchronized (mutex)
                     mutex.wait();
            catch (Exception e)
               e.printStackTrace();
      }.start();
      // This is a simple code used to simulate normal object creation until the GC call
      for (int i = 1;;i++)
         new Object();
         if ((i % 1000) == 0)
            System.out.println(i);
      }NB: This issue has been occured on RHEL 5.3 et JVM Sun 1.6 u26 (64 bit)
Thanks for helping,
BR,
Nicolas

Answer 1

essobedo wrote:
I did not understand that from the javadoc by "javadoc on that field" you mean this :
/* Object used to synchronize with the garbage collector. The collector
* must acquire this lock at the beginning of each collection cycle. It is
* therefore critical that any code holding this lock complete as quickly
* as possible, allocate no new objects, and avoid calling user code.
So for you it is normal that the normal thread "http-0.0.0.0-8080-56" holds the lock expected by the "Reference Handler" and 20 other normal threads? If so do you have any idea why it works like this? Is-it how the GC works internally?if those threads are doing garbage collection work, then yes, they will be holding that lock (which is exactly what the javadoc says).