Portal EHP1 SP07 Patch causes app server jcontrol.exe to not start.....

Question

I just finished patching our test portal system.
The database\message server which also has an app server on it starts up and runs just fine. The problem we are having is the second app server which is on a different windows host will not start (jcontrol.exe stays in a gray status).
Here is the developer trace errors for the second app server:
[Thr 5136] Wed Dec 08 06:56:19 2010
[Thr 5136] *** ERROR => invalid return code of process [bootstrap] (exitcode = 66) [jstartxx.c   1642]
[Thr 5136] JControlExecuteBootstrap: error executing bootstrap node [bootstrap] (rc = 66)
[Thr 5136] JControlCloseProgram: started (exitcode = 66)
[Thr 5136] JControlCloseProgram: good bye... (exitcode = 66)
Here is jvm_bootstrap.out log file located at \usr\sap\xxx\j00\work:
Bootstrap MODE:
<INSTANCE GLOBALS>
determined by parameter [ID0067689].
Exception occurred:
com.sap.engine.bootstrap.SynchronizationException: Database initialization failed! Check database properties!
     at com.sap.engine.bootstrap.Bootstrap.initDatabaseConnection(Bootstrap.java:422)
     at com.sap.engine.bootstrap.Bootstrap.<init>(Bootstrap.java:144)
     at com.sap.engine.bootstrap.Bootstrap.main(Bootstrap.java:814)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:331)
     at com.sap.engine.offline.OfflineToolStart.main(OfflineToolStart.java:81)
==[ Caused by: ]==----
com.sap.engine.frame.core.configuration.ConfigurationException: Error while connecting to DB.
     at com.sap.engine.core.configuration.impl.persistence.rdbms.DBConnectionPool.createConnection(DBConnectionPool.java:360)
     at com.sap.engine.core.configuration.impl.persistence.rdbms.DBConnectionPool.<init>(DBConnectionPool.java:125)
     at com.sap.engine.core.configuration.impl.persistence.rdbms.PersistenceHandler.<init>(PersistenceHandler.java:38)
     at com.sap.engine.core.configuration.impl.cache.ConfigurationCache.<init>(ConfigurationCache.java:149)
     at com.sap.engine.core.configuration.bootstrap.ConfigurationManagerBootstrapImpl.init(ConfigurationManagerBootstrapImpl.java:236)
     at com.sap.engine.core.configuration.bootstrap.ConfigurationManagerBootstrapImpl.<init>(ConfigurationManagerBootstrapImpl.java:49)
     at com.sap.engine.bootstrap.Synchronizer.<init>(Synchronizer.java:60)
     at com.sap.engine.bootstrap.Bootstrap.initDatabaseConnection(Bootstrap.java:419)
     at com.sap.engine.bootstrap.Bootstrap.<init>(Bootstrap.java:144)
     at com.sap.engine.bootstrap.Bootstrap.main(Bootstrap.java:814)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:331)
     at com.sap.engine.offline.OfflineToolStart.main(OfflineToolStart.java:81)
Caused by: com.sap.sql.log.OpenSQLException: Could not load class com.microsoft.sqlserver.jdbc.SQLServerDriver.
     at com.sap.sql.log.Syslog.createAndLogOpenSQLException(Syslog.java:106)
     at com.sap.sql.jdbc.NativeConnectionFactory.createNativeConnection(NativeConnectionFactory.java:143)
     at com.sap.sql.connect.OpenSQLDataSourceImpl.createPooledConnection(OpenSQLDataSourceImpl.java:608)
     at com.sap.sql.connect.OpenSQLDataSourceImpl.getPooledConnection(OpenSQLDataSourceImpl.java:285)
     at com.sap.engine.core.configuration.impl.persistence.rdbms.DBConnectionPool.createConnection(DBConnectionPool.java:302)
     ... 14 more
Caused by: java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver
     at com.sap.engine.offline.FileClassLoader.findClass(FileClassLoader.java:691)
     at com.sap.engine.offline.FileClassLoader.loadClass(FileClassLoader.java:600)
     at com.sap.engine.offline.FileClassLoader.loadClass(FileClassLoader.java:578)
     at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:303)
     at java.lang.Class.forName0(Native Method)
     at java.lang.Class.forName(Class.java:141)
     at com.sap.sql.jdbc.NativeConnectionFactory.createNativeConnection(NativeConnectionFactory.java:134)
     ... 17 more
[Bootstrap module]> Problem occurred while performing synchronization.
Any ideas?
Thank you,
Neil

Answer 1

Neil,
Thanks for your persistence and for reporting how you eventually solved it.  I had precisely the same problem after upgrading our production Portal from 7.00 to 7.01.  I didn't have problems in dev or test, because we don't have multiple app servers in those environments, just a central instance.  Now it's GoLive weekend and I've been pulling out my hair trying to get the app servers to work after the EHPI process is done.  I had to do just what you suggested, about editing the bootstrap configuration manually, as changes in the config tool didn't seem to have any effect on jcontrol.  I got that solved using your solution, but that led to nearly the same problem occurring for the individual dispatcher and server nodes, once jcontrol was started.  It took a while and lot of experimenting, but eventually I figured out that they were pulling the wrong jdbc driver from their individual configuration.manager properties (accessed via config tool).  The central instance was correctly updated by the EHPI program, but not the app servers.  Annoying!
Makes me wonder if I was supposed to leave the app servers running during the 'downtime' phase of the EHPI.  The manual, and the EHPI program, tell you at the start of the phase "shut down all dialog instances now", but then there's no indication of how these significant changes are supposed to be replicated to the app servers.  Several times I nearly escalated to a Very High priority Customer Message, but I kept unraveling the thread until I solved it, and your message here was the clue that put me on the right track.
Thanks,
Matt