Problems with iWS6.0 SP4 after JDK change / upgrade

Question

Solaris8 SPARC
iPlanet 6.0 WebServer SP4
I recently upgraded from JDK 1.3.1_02 to JDK 1.3.1_07 .
In both cases .... the JDK was installed at /usr/j2se.
iPlanet 6.0 SP4 was already using JDK 1.3.1_02 before ( /usr/j2se ).
Now after the JDK upgrade, in an attempt to restart the admserv. I am getting the exception below:
iPlanet-WebServer-Enterprise/6.0SP4 B07/17/2002 14:04
warning: daemon is running as super-user
[LS ls1] http://soft2, port 8888 ready to accept requests
java.lang.NoClassDefFoundError
at java.security.Policy$1.run(Policy.java:144)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.Policy.getPolicyNoCheck(Policy.java:141)
at java.security.SecureClassLoader.getPermissions(SecureClassLoader.java:133)
at java.net.URLClassLoader.getPermissions(URLClassLoader.java:415)
at sun.misc.Launcher$AppClassLoader.getPermissions(Launcher.java:289)
at java.security.SecureClassLoader.getProtectionDomain(SecureClassLoader.java:157)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:106)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:243)
at java.net.URLClassLoader.access$100(URLClassLoader.java:51)
at java.net.URLClassLoader$1.run(URLClassLoader.java:190)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:183)
at java.lang.ClassLoader.loadClass(ClassLoader.java:294)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:281)
at java.lang.ClassLoader.loadClass(ClassLoader.java:250)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:310)
The errors logs say:
[04/Apr/2003:18:36:58] info (20837): successful server startup
[04/Apr/2003:18:36:58] info (20837): iPlanet-WebServer-Enterprise/6.0SP4 B07/17/2002 14:04
[04/Apr/2003:18:36:59] info (20838): Installing a new configuration
[04/Apr/2003:18:36:59] info (20838): [LS ls1] http://soft2, port 8888 ready to accept requests
[04/Apr/2003:18:36:59] info (20838): A new configuration was successfully installed
[04/Apr/2003:18:37:00] failure (20838): Internal error: unable to locate: com/iplanet/server/http/util/LogUtil
[04/Apr/2003:18:37:00] failure (20838): Internal error: Unable to initialize utility classes
[04/Apr/2003:18:37:00] failure (20838): Late initialization failed: Error running init function(late) NSServletLateInit: unknown error
The class com/iplanet/server/http/util/LogUtil is in NSJavaUtil.jar ... which is included in the CLASSPATH in the start-jvm file, show below:
NSES_JDK=/usr/j2se; export NSES_JDK
NSES_JRE=${NSES_JDK}/jre; export NSES_JRE
NSES_JRE_RUNTIME_LIBPATH=${NSES_JRE}/lib/sparc:${NSES_JRE}/lib/sparc/server:${NSES_JRE}/lib/sparc/classic:${NSES_JRE}/lib/sparc/native_thread
s; export NSES_JRE_RUNTIME_LIBPATH
NSES_JDK_RUNTIME_CLASSPATH=${NSES_JRE}/lib/ext/iiimp.jar:${NSES_JRE}/lib/i18n.jar:${NSES_JRE}/lib/rt.jar:${NSES_JDK}/lib/tools.jar:${NSES_JDK
}/lib/dt.jar; export NSES_JDK_RUNTIME_CLASSPATH
# DO NOT CHANGE ANYTHING BELOW THIS LINE
NSES_SERVER_HOME=/usr/iplanet/servers; export NSES_SERVER_HOME
NSES_CLASSPATH=${NSES_SERVER_HOME}/bin/https/jar/NSServletLayer.jar:${NSES_SERVER_HOME}/bin/https/jar/NSJavaUtil.jar:${NSES_SERVER_HOME}/bin/
https/jar/AdminNativeUtil.jar:${NSES_SERVER_HOME}/bin/https/jar/NSJavaMiscUtil.jar:${NSES_SERVER_HOME}/bin/https/jar/servlet.jar:${NSES_SERVE
R_HOME}/bin/https/jar/servlet-2.3-filters-api.jar:${NSES_SERVER_HOME}/bin/https/jar/jsp092.jar:${NSES_SERVER_HOME}/bin/https/jar/jaxp.jar:${N
SES_SERVER_HOME}/bin/https/jar/crimson.jar:${NSES_SERVER_HOME}/bin/https/jar/xalan.jar:${NSES_SERVER_HOME}/bin/https/jar/jspengine.jar; expor
t NSES_CLASSPATH
LD_LIBRARY_PATH=${NSES_JRE_RUNTIME_LIBPATH}:${LD_LIBRARY_PATH}; export LD_LIBRARY_PATH
...and the JAR file itself exists:
soft2:/usr/iplanet/servers/https-admserv >ls -alt /usr/iplanet/servers/bin/https/jar/NS*
-rw-r--r-- 1 root other 191337 Jul 17 2002 /usr/iplanet/servers/bin/https/jar/NSServletLayer.jar
-rw-r--r-- 1 root other 2856 Jul 17 2002 /usr/iplanet/servers/bin/https/jar/NSJavaMiscUtil.jar
-rw-r--r-- 1 root other 29941 Jul 17 2002 /usr/iplanet/servers/bin/https/jar/NSJavaUtil.jar
Here I show that the class iPlanet is complaining is in the NSJavaUtil.jar file:
soft2:/usr/iplanet/servers/https-admserv >jar tvf /usr/iplanet/servers/bin/https/jar/NSJavaUtil.jar | grep Log
3403 Wed Jul 17 13:54:22 GMT+10:00 2002 com/iplanet/server/http/util/LogUtil.class
502 Wed Jul 17 13:54:22 GMT+10:00 2002 com/iplanet/server/http/util/LogUtil$LogWriter.class
If I modify the start-jvm file so that NSES_JDK points to an older version of the JDK ( /usr/Solaris_JDK_1.2.2_07 ) ... or a newer one ( /usr/j2sdk1.4.0_02 ) ... then the admserv instance starts up.
Having said that, in an effort to try to replicate the problem, I tried the same JDK upgrade on another Solaris8 SPARC installation, also with iPlanet 6.0 SP4. That one worked fine with any problems. The admserv instance started fine after the upgrade.
The only different thing that I did was:
1) I first told iPlanet6 to point to a different JDK location ( JDK 1.3.1_03 in /usr/j2sdk1_3_1_03 ) before the upgrade ....
2) restarted the admserv instance ...
3) install JDK 1.3.1_07 ( /usr/j2se )
4) told iplanet6 to point to the JDK 1.3.1_07 ( /usr/j2se )
5) restart the admserv instance
Why would it not work with one installation and not the other ?

Answer 1

This maybe the final clue. I did a truss via:
truss -f ./start -start 2&> err
... and this is what I get just before the NoClassDefFoundError exception ( look at the SIGSEGV ):
22780: lwp_mutex_lock(0xFEBC5558) = 0
22780: lwp_sema_post(0xFCE03E30) = 0
22780: lwp_sema_wait(0xFCE03E30) = 0
22780: lwp_mutex_wakeup(0xFEBC5558) = 0
22780: lwp_mutex_lock(0xFEBC5558) = 0
22780: Incurred fault #6, FLTBOUNDS %pc = 0xF900A8FC
22780: siginfo: SIGSEGV SEGV_ACCERR addr=0xFE427F68
22780: Received signal #11, SIGSEGV [caught]
22780: siginfo: SIGSEGV SEGV_ACCERR addr=0xFE427F68
22780: sigprocmask(SIG_SETMASK, 0xFEBBEFE8, 0x00000000) = 0
22780: mprotect(0xFE424000, 16384, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
22780: sigprocmask(SIG_SETMASK, 0xFEBCAD68, 0x00000000) = 0
22780: setcontext(0xFE42B938)
22780: time() = 1049546821
22780: poll(0xFD007A40, 0, 10) = 0
22780: getpid() = 22780 [22778]
22780: write(6, " [ 0 5 / A p r / 2 0 0 3".., 111)     = 111
22780: lwp_sema_post(0xFCE03E30) = 0
22780: lwp_sema_wait(0xFCE03E30) = 0
22780: lwp_mutex_wakeup(0xFEBC5558) = 0
22780: lwp_mutex_lock(0xFEBC5558) = 0
22780: getrlimit(RLIMIT_STACK, 0xFFBEAFAC) = 0
22780: mprotect(0xFFB72000, 16384, PROT_NONE) = 0
22780: mprotect(0xFFB70000, 8192, PROT_NONE) = 0
22780: poll(0xFD007A40, 0, 10) = 0
22780: brk(0x004BB7F0) = 0
22780: brk(0x004BD7F0) = 0
22780: time() = 1049546821
22780: getpid() = 22780 [22778]
22780: write(6, " [ 0 5 / A p r / 2 0 0 3".., 93)      = 93
java.lang.NoClassDefFoundError22780: write(2, " j a v a . l a n g . N o".., 30) = 30