WebLogic interrupts in two box cluster

Question

          News group: weblogic.developer.interest.clustering
          Subject: WebLogic interrupts in two box cluster
          Hi,
          We have a production environment configured as a single cluster of four WebLogic
          servers using the Apache Proxy plugin. Two of the servers are located here and
          two are located across town (primary and backup) on Sun e10K boxes.
          In our testing environment we have a single cluster of two WebLogic servers running
          on a single box.
          For some reason when we are running in the production environment processing just
          stops (for about 33 seconds but sometimes over 2 minutes) and causes JTA timeouts.
          I have observed this during the create() of a stateless session bean, but I have
          been "told" that it happens randomly. This has NEVER happend in the test environment.
          Some of the session EJB create() methods are calling 4 other EJBs. I know that
          JNDI look ups are costly (average between 320ms and 550ms) but that doesn't get
          us to 33 seconds.
          Now for the constraints:
          We are not allowed to have an initial pool of EJBs (don't ask).
          We are not allowed to have local interfaces on the EJBs (don't ask).
          I am not allowed to see the source code (don't ask) to find out if a UserTransaction
          has been set.
          Some questions:
          If a UserTransaction has not been set, will WebLogic ALWAYS create the the other
          EJBs that the stateless session bean calls on the same server (collocated) instance
          or will they go across the wire to the next available server (because of the remote
          interfaces and the home objects having the load balancing logic)? (Note: the
          EJB descriptor may have transaction delineation defined on the EJBs participating
          in the transaction and the configuration we are using only provides round robin
          load balancing).
          Will the home objects requested by the session bean be created on the server the
          code is currently running (collocated) on or will any server with the distributed
          JNDI tree be allowed to provide the home object?
          If the above could happen, then paragraph 2 on page 4-10 of "Understanding Object
          Clustering" (Bea WebLogic Server 6.0) might come in to play where we have remote
          replicas adding to network overhead for the duration of the transaction. Since
          the processors on all the boxes appear to be more or less idle and there is loads
          of RAM available... Also I was "told" this only happens under load (100 simultaneous
          hits).
          Could we be falling into a convoying condition? Does the whole cluster stop when
          this condition happens or does just the single instance stop? If we have one
          server participating in the same transaction as another server would that stop
          the whole cluster? If we have 4 EJBs and load balancing decides all servers are
          participating in the same transaction (round robin on 4 servers gets all of them
          in the game), would that stop the whole cluster while data I/O flows across the
          wire?
          When an initial instance of an EJB is created (initial pool size), does the entire
          cluster get informed or does each server independantly manage its own pool? What
          about EJB instances when replica-aware EJBObject stubs are used. Documentation
          states that the stub is free to route any call to any server that hosts the bean,
          could this also bring more than one server into the same transaction?
          Other groups are looking into HTTP Session replication as a possible cause...
          Thanks for any thoughts,
          Ian
          

Answer 1

          Some results:
          The script directly targeted a single weblogic instance within the OT cluster,
          presumably taking the Alteon switch and (perhaps? probably?) the clustering out
          of the equation.
          The test was the bro regle script with 1 concurrent user (same test run in OT
          on October 14 overnight). The results below show weblogic.transaction.internal.TimedOutException
          caught by this application.
          2003-10-15 17:35:52,859
          2003-10-15 18:02:00,433
          2003-10-15 18:27:42,110
          2003-10-15 19:02:22,767
          2003-10-15 19:28:05,193
          2003-10-15 20:28:28,584
          2003-10-15 21:03:07,879
          2003-10-15 21:28:50,622
          2003-10-15 22:03:29,683
          2003-10-15 22:28:51,065
          these timeouts are falling roughly 25 minutes apart.
          Does this help at all?
          Ian
          "Ian Douglas" <[email protected]> wrote:
          >
          >Thanks,
          >
          >Will do.
          >
          >This may be OS related as we have found, that with a single automated
          >user, that
          >it interrupts every 60 minutes for the duration of some process.
          >
          >Ian
          >
          >"Sree Bodapati" <[email protected]> wrote:
          >>Take multiple threaddumps when server appears to have stopped processing.
          >>Threaddumps would tell you whats happening.
          >>
          >>
          >>
          >>
          >>"Ian Douglas" <[email protected]> wrote in message
          >>news:[email protected]...
          >>>
          >>> Any news on this one? A 30 second time out is pretty long...
          >>>
          >>> Ian
          >>>
          >>> "Ian Douglas" <[email protected]> wrote:
          >>> >
          >>> >News group: weblogic.developer.interest.clustering
          >>> >
          >>> >Subject: WebLogic interrupts in two box cluster
          >>> >
          >>> >Hi,
          >>> >
          >>> >We have a production environment configured as a single cluster of
          >>four
          >>> >WebLogic
          >>> >servers using the Apache Proxy plugin. Two of the servers are located
          >>> >here and
          >>> >two are located across town (primary and backup) on Sun e10K boxes.
          >>> >
          >>> >
          >>> >In our testing environment we have a single cluster of two WebLogic
          >>servers
          >>> >running
          >>> >on a single box.
          >>> >
          >>> >For some reason when we are running in the production environment
          >>processing
          >>> >just
          >>> >stops (for about 33 seconds but sometimes over 2 minutes) and causes
          >>> >JTA timeouts.
          >>> > I have observed this during the create() of a stateless session
          >bean,
          >>> >but I have
          >>> >been "told" that it happens randomly. This has NEVER happend in
          >the
          >>> >test environment.
          >>> >
          >>> >Some of the session EJB create() methods are calling 4 other EJBs.
          >> I
          >>> >know that
          >>> >JNDI look ups are costly (average between 320ms and 550ms) but that
          >>doesn't
          >>> >get
          >>> >us to 33 seconds.
          >>> >
          >>> >Now for the constraints:
          >>> >We are not allowed to have an initial pool of EJBs (don't ask).
          >>> >We are not allowed to have local interfaces on the EJBs (don't ask).
          >>> >
          >>> >I am not allowed to see the source code (don't ask) to find out if
          >>a
          >>> >UserTransaction
          >>> >has been set.
          >>> >
          >>> >Some questions:
          >>> >If a UserTransaction has not been set, will WebLogic ALWAYS create
          >>the
          >>> >the other
          >>> >EJBs that the stateless session bean calls on the same server
          >>(collocated)
          >>> >instance
          >>> >or will they go across the wire to the next available server (because
          >>> >of the remote
          >>> >interfaces and the home objects having the load balancing logic)?
          >> (Note:
          >>> >the
          >>> >EJB descriptor may have transaction delineation defined on the EJBs
          >>participating
          >>> >in the transaction and the configuration we are using only provides
          >>round
          >>> >robin
          >>> >load balancing).
          >>> >
          >>> >Will the home objects requested by the session bean be created on
          >>the
          >>> >server the
          >>> >code is currently running (collocated) on or will any server with
          >>the
          >>> >distributed
          >>> >JNDI tree be allowed to provide the home object?
          >>> >
          >>> >If the above could happen, then paragraph 2 on page 4-10 of
          >>"Understanding
          >>> >Object
          >>> >Clustering" (Bea WebLogic Server 6.0) might come in to play where
          >>we
          >>> >have remote
          >>> >replicas adding to network overhead for the duration of the transaction.
          >>> > Since
          >>> >the processors on all the boxes appear to be more or less idle and
          >>there
          >>> >is loads
          >>> >of RAM available... Also I was "told" this only happens under load
          >>(100
          >>> >simultaneous
          >>> >hits).
          >>> >
          >>> >Could we be falling into a convoying condition? Does the whole cluster
          >>> >stop when
          >>> >this condition happens or does just the single instance stop? If
          >>we
          >>> >have one
          >>> >server participating in the same transaction as another server would
          >>> >that stop
          >>> >the whole cluster? If we have 4 EJBs and load balancing decides
          >all
          >>> >servers are
          >>> >participating in the same transaction (round robin on 4 servers gets
          >>> >all of them
          >>> >in the game), would that stop the whole cluster while data I/O flows
          >>> >across the
          >>> >wire?
          >>> >
          >>> >When an initial instance of an EJB is created (initial pool size),
          >>does
          >>> >the entire
          >>> >cluster get informed or does each server independantly manage its
          >>own
          >>> >pool? What
          >>> >about EJB instances when replica-aware EJBObject stubs are used.
          >>Documentation
          >>> >states that the stub is free to route any call to any server that
          >>hosts
          >>> >the bean,
          >>> >could this also bring more than one server into the same transaction?
          >>> >
          >>> >Other groups are looking into HTTP Session replication as a possible
          >>> >cause...
          >>> >
          >>> >Thanks for any thoughts,
          >>> >Ian
          >>>
          >>
          >>
          >