[WLS92] Thread synchronization bottleneck in RJVMManager

Question

All,
First off, apologies for the long post -- there is a lot to get through.
h3. Environment:
Weblogic Server 9.2 MP3 deployed into Solaris 10 / Sparc hosts.
Sun JDK 1.5.0_12 JVM (shipped with WLS)
h3. Problem:
We are experiencing a problem with out WLS 9.2/MP3 (Solaris/Sparc) installation. We have a number of servers each running the same WLS configuration but some with different application: one server is running App A, 4 other servers are running App B (not in a cluster). We are using JMS to transfer messages from App A to App B in a strict failover configuration such that App A will always try and send to App B1, App B2, AppB3 and finally App B4, moving from one to the next if the transfer fails or takes longer than a given timeout (10 seconds). The dispatching happens in separate threads, which I will refer to as 'senders', so we can timeout the transfer attempt.
In the 'happy day' scenario where all four App B servers are up and running, everything works as expected.
Problems start to surface when we test the failover configurations.
In scenarios where the hosts are available but WLS is in either not running or in Admin mode, the failover proceeds as expected.
In scenarios where the hosts are unavailable (hardware is powered down, network is disrupted etc.), the failover does not proceed as expected. In a scenario where App B1 is down but the other three are available for example, the following are observed:
1. the time taken for the underlying socket connection to App B1 to timeout is very long (~7.5 minutes).
2. the connections to the other hosts appear to block until after the socket timeout to App B1 occurs.
The relevant section of the code is:
weblogic.security.Security.runAs(new Subject(), new PrivilegedAction<Void>() {
     public Void run() {
          InitialContext context = null;
          try {
               * During server-to-server sending of messages we must establish and maintain
               * a new context that will contain the security information established by
               * the initial context object. This has to be done per thread, hence why it
               * is applied here to the dispatcher. The context will be closed when
               * the thread is returned.
               final Hashtable<?,?> environment = getEnvironment();
               try {
                    if (log.isDebugEnabled()) {
                         log.debug(format("run(): Sender %d Creating initial context; env=%s", id, environment));
                    context = new InitialContext(environment);
                    if (log.isDebugEnabled()) {
                         log.debug(format("run(): Sender %d Created initial context in %d ms", id, System.currentTimeMillis() - start));
                    runWithContext();
               } catch (NamingException e) {
                    log.error(format("run(): Sender %d cannot create initial context", id), e);
          } finally {
               try {                           
                    if (context != null) {
                         if (log.isDebugEnabled()) {
                              log.debug(format("run(): Sender %d closing initial context", id, context));
                         context.close();
               } catch (NamingException e) {
                    log.error(format("run(): Sender %d cannot close initial context", id), e);
          return null;
Thread stacks for two sample threads in this scenario are as follows. Thread '20' is the sender attempting to connect to App B1, Thread '8' is the sender attempting to connect to App B2 and is representative of a number of other threads all attempting to connect to known good hosts.
Daemon Thread [[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'] (Suspended)     
     PlainSocketImpl.socketConnect(InetAddress, int, int) line: not available [native method]     
     SocksSocketImpl(PlainSocketImpl).doConnect(InetAddress, int, int) line: 333     
     SocksSocketImpl(PlainSocketImpl).connectToAddress(InetAddress, int, int) line: 195     
     SocksSocketImpl(PlainSocketImpl).connect(SocketAddress, int) line: 182     
     SocksSocketImpl.connect(SocketAddress, int) line: 366     
     SSLSocketImpl(Socket).connect(SocketAddress, int) line: 519     
     SSLSocketImpl(Socket).connect(SocketAddress) line: 469     
     SSLSocketImpl(Socket).<init>(SocketAddress, SocketAddress, boolean) line: 366     
     SSLSocketImpl(Socket).<init>(InetAddress, int, InetAddress, int) line: 266     
     SSLSocketImpl(SSLSocket).<init>(InetAddress, int, InetAddress, int) line: 195     
     SSLSocketImpl.<init>(TLSSystem, InetAddress, int, InetAddress, int) line: not available     
     SSLSocketFactoryImpl.createSocket(InetAddress, int, InetAddress, int) line: not available     
     ChannelSSLSocketFactory.createSocket(InetAddress, int) line: 77     
     MuxableSocketT3S(AbstractMuxableSocket).createSocket(InetAddress, int, int) line: 207     
     MuxableSocketT3S(MuxableSocketT3).newSocketWithRetry(InetAddress, int, int) line: 252     
     MuxableSocketT3S(MuxableSocketT3).connect(InetAddress, int, int) line: 421     
     MuxableSocketT3S.createConnection(InetAddress, int, ServerChannel, JVMID, int) line: 79     
     ConnectionManager.createConnection(Protocol, InetAddress, int, ServerChannel, JVMID, int) line: 1749     
     ConnectionManagerServer(ConnectionManager).findOrCreateConnection(ServerChannel, JVMID, int) line: 1410     
     ConnectionManagerServer(ConnectionManager).bootstrap(JVMID, ServerChannel, int) line: 448     
     ConnectionManagerServer(ConnectionManager).bootstrap(InetAddress, int, ServerChannel, int) line: 326     
     RJVMManager.findOrCreateRemoteInternal(InetAddress, int, String, String, int) line: 261     
     RJVMManager.findOrCreate(InetAddress, int, String, String, int) line: 204     
     RJVMFinder.findOrCreateRemoteServer(InetAddress, int, String, int) line: 226     
     RJVMFinder.findOrCreate(boolean, String, HostID, int) line: 189     
     ServerURL.findOrCreateRJVM(boolean, String, HostID, int) line: 154     
     WLInitialContextFactoryDelegate$1.run() line: 342     
     AuthenticatedSubject.doAs(AbstractSubject, PrivilegedExceptionAction) line: 363     
     SecurityManager.runAs(AuthenticatedSubject, AuthenticatedSubject, PrivilegedExceptionAction) line: 147     
     WLInitialContextFactoryDelegate.getInitialContext(Environment, String, HostID) line: 337     
     Environment.getContext(String, HostID) line: 307     
     Environment.getContext(String) line: 277     
     WLInitialContextFactory.getInitialContext(Hashtable) line: 117     
     NamingManager.getInitialContext(Hashtable<?,?>) line: 667     
     InitialContext.getDefaultInitCtx() line: 247     
     InitialContext.init(Hashtable<?,?>) line: 223     
     InitialContext.<init>(Hashtable<?,?>) line: 197     
     FailoverMessageSender$Sender$1.run() line: 475     
     FailoverMessageSender$Sender$1.run() line: 458     
     AuthenticatedSubject.doAs(AbstractSubject, PrivilegedAction) line: 321     
     SecurityManager.runAs(AuthenticatedSubject, AuthenticatedSubject, PrivilegedAction) line: 121     
     Security.runAs(Subject, PrivilegedAction) line: 41     
     FailoverMessageSender$Sender.run() line: 457     
     DelegatingWork.run() line: 61     
     J2EEWorkManager$WorkWithListener.run() line: 259     
     ExecuteThread.execute(Runnable) line: 209     
     ExecuteThread.run() line: 181     
Daemon Thread [[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'] (Suspended)     
     RJVMManager.findOrCreateRemoteInternal(InetAddress, int, String, String, int) line: 252     
     RJVMManager.findOrCreate(InetAddress, int, String, String, int) line: 204     
     RJVMFinder.findOrCreateRemoteServer(InetAddress, int, String, int) line: 226     
     RJVMFinder.findOrCreate(boolean, String, HostID, int) line: 189     
     ServerURL.findOrCreateRJVM(boolean, String, HostID, int) line: 154     
     WLInitialContextFactoryDelegate$1.run() line: 342     
     AuthenticatedSubject.doAs(AbstractSubject, PrivilegedExceptionAction) line: 363     
     SecurityManager.runAs(AuthenticatedSubject, AuthenticatedSubject, PrivilegedExceptionAction) line: 147     
     WLInitialContextFactoryDelegate.getInitialContext(Environment, String, HostID) line: 337     
     Environment.getContext(String, HostID) line: 307     
     Environment.getContext(String) line: 277     
     WLInitialContextFactory.getInitialContext(Hashtable) line: 117     
     NamingManager.getInitialContext(Hashtable<?,?>) line: 667     
     InitialContext.getDefaultInitCtx() line: 247     
     InitialContext.init(Hashtable<?,?>) line: 223     
     InitialContext.<init>(Hashtable<?,?>) line: 197     
     FailoverMessageSender$Sender$1.run() line: 475     
     FailoverMessageSender$Sender$1.run() line: 458     
     AuthenticatedSubject.doAs(AbstractSubject, PrivilegedAction) line: 321     
     SecurityManager.runAs(AuthenticatedSubject, AuthenticatedSubject, PrivilegedAction) line: 121     
     Security.runAs(Subject, PrivilegedAction) line: 41     
     FailoverMessageSender$Sender.run() line: 457     
     DelegatingWork.run() line: 61     
     J2EEWorkManager$WorkWithListener.run() line: 259     
     ExecuteThread.execute(Runnable) line: 209     
     ExecuteThread.run() line: 181     
h3. Conclusion:
All of the threads are jamming up in RJVMManager.findOrCreateRemoteInternal(..) behind the one thread that is actually attempting to make a connection.
A quick disassemby of the offending method class shows the following snippet:
/* */ private RJVM findOrCreateRemoteInternal(InetAddress paramInetAddress, int paramInt1, String paramString1, String paramString2, int paramInt2)
/* */ throws UnrecoverableConnectException, IOException
/* 246 */ RJVM localRJVM = findExisting(paramInetAddress, paramInt1, (ServerChannel)localObject1);
/* 248 */ if (localRJVM != null) return localRJVM;
/* 250 */ synchronized (this.bootstrapLock)
/* 252 */ localRJVM = findExisting(paramInetAddress, paramInt1, (ServerChannel)localObject1);
/* 253 */ if (localRJVM != null) return localRJVM;
/* 256 */ if ((Kernel.DEBUG) && (debugConnection.isDebugEnabled())) {
/* 257 */ RJVMLogger.logDebug("Bootstrapping connection to: '" + paramInetAddress + ":" + paramInt1 + "' using: '" + localProtocol + "'");
/* 260 */ ConnectionManager localConnectionManager = ConnectionManager.create(null);
/* 261 */ return record(localConnectionManager.bootstrap(paramInetAddress, paramInt1, (ServerChannel)localObject1, paramInt2), paramInetAddress);
Now the problem becomes clear. I suspect that the reason for the synchronized block on line 250 is to prevent two threads attempting the relatively expensive operation on line 261 (see lines 248-250 and 252-253). However, the flaw here is that the synchronization blocks all threads irrespective of which InetAddress (param 1) the thread is attempting to connect to. My personal opinion is that the monitor used for thread synchronization should be timed to the host (InetAddress) for the connection, which incidentally is used in the record(..) method anyway.
The problem is exacerbated by the very long network connection timeout.
h3. Solutions:
We've tried setting the sun.net.defaultConnectTimeout JVM property to try and reduce the impact but haven't seen any noticeable improvement.
We've tried specifying the weblogic.jndi.clientTimeout and weblogic.jndi.requestTimeout values when creating the InitialContext. For some reason specifying these properties caused all connection attempts to fail -- even the happy day scenarios.
h3. Final Questions:
Is this a bug? If so, what do I need to do to get this fixed urgently?
Is this a bug other people have run into? If so, how was it resolved?
Any other suggestions?
Thanks!
Edited by: user11373704 on 10-Jul-2009 08:52

Answer 1

Perhaps the relevant timeouts are socket/network related in this case.
To detect "silent" network failures, WebLogic uses a heartbeat mechanism, but I'm told that one should not tune the related settings too aggressively. I think this is tunable on a per server basis -- at least for the default network channels. Specifically there's heartbeat interval and period length on the console at "Server -> Configuration Tab --> Tuning Tab --> Advanced".
There's also "Complete Message Timeout" under "Protocol -> General".
In addition, TCP/IP timeouts are tunable on the operating system itself. I don't know the specifics for Solaris.
Tom