Problems with signed JAR files in JWS/JRE6 environment.

Question

Hello All,
I'm encountering a problem running our desktop application as a Java Web Start deployment in a JRE 6 environment. There were never any problems when running the same application as a JWS deployment in JRE 1.4, or 5, environments. There are also currently no problems in a JRE 6 environment when running the application as a standard desktop application.
The problem which I am having has nothing to do with launching the application. But for good measure, I verified the JNLP file with JaNeLA. A couple things we out of order, which I addressed to make JaNeLA happy, but my problem still persists. Here is my JNLP file (anonymized to protect the innocent):
TS: 2010-10-18 17:04:46
<?xml version="1.0" encoding="UTF-8"?>
<jnlp codebase="$$codebase" href="$$name">
     <information>
          <title>Acme Desktop</title>
          <vendor>Acme Corporation</vendor>
          <homepage href="http://www.acme.com/"/>
          <description>Acme Client for Acme Server</description>
          <description kind="tooltip">Acme Client for Acme Server</description>
          <icon href="desktop.gif"/>
          <offline-allowed/>
     </information>
     <security>
          <all-permissions/>
     </security>
     <resources>
          <j2se version="1.5+"/>
          <jar href="acmedesktop.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/antlr-2.7.2.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/backport-util-concurrent.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/commons-codec-1.3.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/commons-httpclient.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/commons-logging.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/acmeapi.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/HelpJavaDT.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/HelpJavaDT_es.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/jacorb.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/Multivalent.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/slf4j-api-1.5.6.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/slf4j-jdk14-1.5.6.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/snow.jar" download="lazy" version="8.00.01.00+"/>
          <jar href="lib/AcmeTMClient.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/xercesImpl.jar" download="eager" version="8.00.01.00+"/>
          <jar href="lib/xml-apis.jar" download="eager" version="8.00.01.00+"/>
          <extension name="installer" href="desktopInstaller.jnlp" />
          <extension name="Java Help" href="help.jnlp"/>
          <property name="java.library.path" value="./lib"/>
          <property name="admin" value="false"/>
          <property name="webstart" value="true"/>          
          <!-- The following two lines are for SSO implementation only
          <property name="urladdress" value="http://localhost:8080/AcmeDesktop/servlet/AcmeServlet"/>
          <property name="cookiespec" value="RFC2109"/>
          -->          
     </resources>
     <resources os="Windows">
          <nativelib href="lib/jniWin32.jar" version="8.00.01.00+"/>
     </resources>
     <application-desc main-class="desktop"/>     
</jnlp>-----
When running as a JWS deployment, on JRE 6, the application will be functioning normally for a little while, and then suddenly the following exception is thrown, and the current operation fails because the class in question cannot be accessed:
java.lang.SecurityException: class "acmeapi.communication.CDocImpl"'s signer information does not match signer information of other classes in the same package
     at java.lang.ClassLoader.checkCerts(ClassLoader.java:807)
     at java.lang.ClassLoader.preDefineClass(ClassLoader.java:488)
     at java.lang.ClassLoader.defineClassCond(ClassLoader.java:626)
     at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
     at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
     at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
     at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
     at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
     at java.security.AccessController.doPrivileged(Native Method)
     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
     at com.sun.jnlp.JNLPClassLoader.findClass(JNLPClassLoader.java:288)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
     at acmeapi.common.CDoc.getAnnotationsInfo(CDoc.java:493)
     at acmedesktop.communication.CCommunicationManager.privateGetAnnotations(CCommunicationManager.java:1976)
     at acmedesktop.communication.CCommunicationManager.getAnnotations(CCommunicationManager.java:1828)
     at acmedesktop.annotations.CViewAnnotations.getAnnotations(CViewAnnotations.java:826)
     at acmedesktop.annotations.CViewAnnotations.createView(CViewAnnotations.java:583)
     at acmedesktop.annotations.CViewAnnotations.setData(CViewAnnotations.java:736)
     at acmedesktop.annotations.CViewAnnotations.init(CViewAnnotations.java:205)
     at acmedesktop.hitspanel.CHitsPanel.viewAnnotations(CHitsPanel.java:281)
     at acmedesktop.hitspanel.CHitsTab$3.mousePressed(CHitsTab.java:316)
     at java.awt.AWTEventMulticaster.mousePressed(AWTEventMulticaster.java:263)
     at java.awt.Component.processMouseEvent(Component.java:6260)
     at javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
     at java.awt.Component.processEvent(Component.java:6028)
     at java.awt.Container.processEvent(Container.java:2041)
     at java.awt.Component.dispatchEventImpl(Component.java:4630)
     at java.awt.Container.dispatchEventImpl(Container.java:2099)
     at java.awt.Component.dispatchEvent(Component.java:4460)
     at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4574)
     at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4235)
     at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4168)
     at java.awt.Container.dispatchEventImpl(Container.java:2085)
     at java.awt.Window.dispatchEventImpl(Window.java:2478)
     at java.awt.Component.dispatchEvent(Component.java:4460)
     at java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
     at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
     at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
     at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
     at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
     at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
     at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)-----
The classes of our desktop product are contained within the 'acmedesktop' and 'acmeapi' packages. It requires access to the hard drive of the workstation, and therefore, all jar files included with the application are signed using the following ANT task when compiled:
<signjar keystore="resources/codesigning/keystore.pfx" storetype="pkcs12" storepass="myPassword" alias="myAlias">
     <fileset dir="${jws_dist}/app" includes="*.jar"/>
     <fileset dir="${jws_dist}/app/lib" includes="*.jar" excludes="jhall__V${dt_version}.jar"/>
</signjar>-----
Therefore, all classes, within all jar files, are signed with the same certificate (with the exception of the JavaHelp libraries, which are already signed by Sun - but the class in question attempting to be loaded here is not contained within the JavaHelp jar file anyway). So, the point being, that the exception message stating that the "signer information of the acmeapi.communication.CDocImpl class doesn't match the signer information of other classes in the same package", is simply not correct. All classes within that jar file were signed using the same certificate.
I downloaded the JRE 6 source from dev.java.net and picked through this issue with a debugger. The ClassLoader.checkCerts() method compares the certificate used to sign the current class which is attempting to be loaded, with the certificates which signed all other previously loaded classes within the same package. If they don't match, the exception above is thrown. What is causing the issue is when the checkCerts() method attempts to get the certificates which signed the currently loading class, null is returned. And obviously, comparing null, with an array of the certificates which signed the previously loaded classes, isn't going to match; therefore this exception is thrown.
The checkCerts() method gets the certificates of the currently loading class by calling the java.security.CodeSource.getCertificates() method. Tracing deeper in the debugger, the CodeSource object ultimately gets the certificates from the 'signersRef' member variable of the com.sun.deploy.cache.CachedJarFile class. signerRef is a SoftReference object and can therefore be garbage collected at some point. If it has already been garbage collected, the CachedJarFile class will attempt to retrieve it again from the loaded cache entry by calling com.sun.deploy.cache.MemoryCache.getLoadedResource().
The MemoryCache class maintains the cache entries to the jar files as MemoryCache.CachedResourceReference objects, which subclass WeakReference, and therefore these objects can be garbage collected as well. If the cache entries have also been garbage collected, this leaves the CachedJarFile class with no ability to repopulate the CachedJarFile.signerRef object. Therefore it is completely out of luck getting the certificates which signed the currently loading class, which ultimately causes the above exception.
When the com.sun.deploy.cache.Cache class attempts to retrieve a cache entry using its getCacheEntry() method, it will attempt to get the entry from the MemoryCache class, if null is returned, it will recreate the cache entry and add it back to the MemoryCache. In contrast, when the CachedJarFile class attempts to get a cache entry from the MemoryCache class, if null is returned, it just gives up.
(from com.sun.deploy.cache.CachedJarFile:244)
private CacheEntry getCacheEntry() {
     /* if it was not created by Cache do not search for entry */
     if (resourceURL == null)
          return null;
     CacheEntry ce = (CacheEntry) MemoryCache.getLoadedResource(resourceURL);
     if (ce == null) {
          //This should not happen because CacheEntry should not get collected
          // before CachedJarFile is collected.
          Trace.println("Missing CacheEntry for " + resourceURL + "\n" + ce,
               TraceLevel.CACHE);
     return ce;
When debugging, code execution falls within the code block with the comment stating "This should not happen...", but it is happening in my case.
On an interesting side note, using the jvisualvm.exe tool included with JDK 6, I was able to tell that it seems as though these objects are collected the first time that the JVM allocates more heap space, and then the issue will occur. If I set the initial heap size very large (using -Xms) this issue won't occur at all. But that is kind of a bad solution which I would rather not do, but it is interesting to note for the sake of troubleshooting this issue. The max heap size (-Xmx) is plenty big enough, so the issue is not that we are running out of memory here.
Does anyone have any insight as to what could be causing this? I've searched, and found a couple threads with similar problems but with no clear solutions. It is not just one workstation either, it happens everywhere I deploy the app as a Java Web Start application in a JRE 6 environment. I have been using version 1.6.0_18 on XP, but it seems to happen on any update version of 1.6. Is the fact that the CachedJarFile class doesn't attempt to reload the resource when it can't retrieve it from MemoryCache a bug? I've dug as deep as I can on this and I'm at wits end, does anybody have any ideas?
Thank you
Jake
Edited by: jkc532 on Nov 12, 2010 10:35 AM

Answer 1

jkc532 wrote:
.. Is the fact that the CachedJarFile class doesn't attempt to reload the resource when it can't retrieve it from MemoryCache a bug? From your comprehensive investigation and report, it seems so to me.
..I've dug as deep as I can on this and I'm at wits end, does anybody have any ideas?Just after read the summary I was tired, so I have some understanding of the effort you have already invested in this (the 'wits' you have already spent). I think you should raise a bug report and seek Oracle's response.