Is a cluster proxy a single-point-of-failure?

Question

Our group is planning on configuring a two machine cluster to host
          servlets/jsp's and a single backend app server to host all EJBs and a
          database.
          IIS is going to be configured on each of the two cluster machines with a
          cluster plugin. IIS is being used to optimize performance of static HTTP
          requests. All servlet/jsp request would be forwarded to the Weblogic
          cluster. Resonate's Central Dispatch is also going to be installed on the
          two cluster machines. Central Dispatch is being used to provide HTTP
          request load-balancing and to provide failover in case one of the IIS
          servers fails (because the IIS process fails or the cluster machine it's on
          fails).
          Will this configuration work? I'm most concerned about the failover of the
          IIS cluster proxy. If one of the proxies is managing a sticky session (X),
          what happens when the machine (the proxy is on) dies and we failover to the
          other proxy? Is that proxy going to have any awareness of session X?
          Probably not. The new proxy is probably going to believe this request is
          new and forward the request to a machine which may not host the existing
          primary session. I believe this is an error?
          Is a cluster proxy a single-point-of-failure? Is there any way to avoid
          this? Does the same problem exist if you use Weblogic's HTTP server (as the
          cluster proxy)?
          Thank you.
          Marko.
          

Answer 1

We found our entity bean bottlenecks using JProbe Profiler. It's great for
          watching the application and seeing what methods it spends its time in. We
          found an exeedingly high number of calls to ejbLoad were taking a lot of
          time, probably due to the fact that our EBs don't all have bulk-access
          methods.
          We also had to do some low-level method tracing to watch WebLogic thrash EB
          locks, basically it locks the EB instance every time it is accessed in a
          transaction. Our DBA says that Oracle is seeing a LOT of lock/unlock
          activity also. Since much of our EB data is just configuration information
          we don't want to incur the overhead of Java object locks, excess queries,
          and Oracle row locks just to read some config values. Deadlocks were also a
          major issue because many txns would access the same config data.
          Our data is also very normalized, and also very recursive, so using EBs
          makes it tricky to do joins and recursive SQL queries. It's possible that we
          could get good EB performance using bulk-access methods and multi-table EBs
          that use custom recursive SQL queries, but we'd still have the
          lock-thrashing overhead. Your app may differ, you may not run into these
          problems and EBs may be fine for you.
          If you have a cluster proxy you don't need to use sticky sessions with your
          load balancer. We use sticky sessions at the load-balancer level because we
          don't have a cluster proxy. For our purposes we decided that the minimal
          overhead of hardware ip-sticky session load balancing was more tolerable
          than the overhead of a dog-slow cluster proxy on WebLogic. If you do use the
          proxy then your load balancer can do round-robin or any other algorithm
          amongst all the proxies.
          Marko Milicevic <[email protected]> wrote in message
          news:[email protected]...
          > Sorry Grant. I meant to reply to the newsgroup. I am putting this reply
          > back on the group.
          >
          > Thanks for your observations. I will keep them all in mind.
          > Is there any easy way for me to tell if I am getting acceptable
          performance
          > with our configuration? For example, how do I know if my use of Entity
          > beans is slow? Will I have to do 2 implementations? One implementation
          > using entity beans and anther implementation that replaces all entity use
          > with session beans, then compare the performance?
          >
          > One last question about the cluster proxy. You mentioned that you are
          using
          > Load Director with sticky sessions. We too are planning on using sticky
          > sessions with Central Dispatch. But since the cluster proxy is stateless,
          > does it matter if sticky sessions is used by the load balancer? No matter
          > which cluster proxy the request is directed to (by load balancing) the
          > cluster proxy will in turn redirect the request to the correct machine
          (with
          > the primary session). Is this correct? If I do not have to incur the
          cost
          > of sticky sessions (with the load balancer) I would rather avoid it.
          >
          > Thanks again Grant.
          >
          > Marko.
          > .
          >
          > -----Original Message-----
          > From: Grant Kushida [mailto:[email protected]]
          > Sent: Monday, May 01, 2000 5:16 PM
          > To: Marko Milicevic
          > Subject: RE: Is a cluster proxy a single-point-of-failure?
          >
          >
          > We haven't had too many app server VM crashes, although our web server
          > typically needs to be restarted every day or so due to socket strangeness
          or
          > flat out process hanging. Running 2 app server processes on the same box
          > would help with the VM stuff, but remember to get 2 NICs, because all
          > servers on a cluster need to run on the same port with different IP addrs.
          >
          > We use only stateless session beans and entity beans - we have had a
          number
          > of performance problems with entity beans though so we will be migrating
          > away from them shortly, at least for our configuration-oriented tables.
          > Since each entity (unique row in the database) can only be accessed by one
          > transaction at a time, we ran into many deadlocks. There was also a lot of
          > lock thrashing because of this transaction locking. And of course the
          > performance hit of the naive database synching (read/write for each method
          > call). We're using bean-managed persistence in 4.5.1, so no read-only
          beans
          > for us yet.
          >
          > It's not the servlets that are slower, it's the response time due to the
          > funneling of requests through the ClusterProxy servlet running on a
          WebLogic
          > proxy server. You don't have that configuration so you don't really need
          to
          > worry. Although i have heard about performance issues with the cluster
          proxy
          > on IIS/netscape, we found performance to be just fine with the Netscape
          > proxy.
          >
          > We're currently using no session persistence. I have a philosophical issue
          > with going to vendor-specific servlet extensions that tie us to WebLogic.
          We
          > do the session-sticky load balancing with a Cisco localdirector, meanwhile
          > we are investigating alternative servlet engines (Apache/JRun being the
          > frontrunner). We might set up Apache as our proxy server running the
          > Apache-WL proxy plugin once we migrate up to 5.1, though.
          >
          > > -----Original Message-----
          > > From: Marko Milicevic [mailto:[email protected]]
          > > Sent: Monday, May 01, 2000 1:08 PM
          > > To: Grant Kushida
          > > Subject: Re: Is a cluster proxy a single-point-of-failure?
          > >
          > >
          > > Thanks for the info Grant.
          > >
          > > That is good news. I was worried that the proxy maintained
          > > state, but since
          > > it is all in the cookie, then I guess we are ok.
          > >
          > > As for the app server, you are right. It is a single point
          > > of failure, but
          > > the machine is a beast (HP/9000 N-class) with hardware
          > > redundancy up the
          > > yin-yang. We were unsure how much benefit we would get if we
          > > clustered
          > > beans. There seems to be a lot of overhead associated with
          > > clustered entity
          > > beans since every bean read includes a synch with the
          > > database, and there is
          > > no fail over support. Stateful session beans are not load
          > > balanced and do
          > > not support fail over. There seems to be real benefit for
          > > only stateless
          > > beans and read-only entities. Neither of which we have many
          > > of. We felt
          > > that we would probably get better performance by locating all
          > > of our beans
          > > on the same box as the data source. We are considering creating a two
          > > instance cluster within the single app server box to protect
          > > against a VM
          > > crash. What do you think? Do you recommend a different
          > > configuration?
          > >
          > > Thanks for the servlet performance tip. So you are saying
          > > that running
          > > servlets without clustering is 6-7x faster than with
          > > clustering? Are you
          > > using in-memory state replication for the session? Is this
          > > performance
          > > behavior under 4.5, 5.1, or both? We are planning on
          > > implementing under
          > > 5.1.
          > >
          > > Thanks again Grant.
          > >
          > > Marko.
          > > .
          >
          >
          > Grant Kushida <[email protected]> wrote in message
          > news:[email protected]...
          > > Seems like you'll be OK as far as session clustering goes. The Cluster
          > > proxies running on your IIS servers are pretty dumb - they just analyze
          > the
          > > cookie and determine the primary/secondary IP addresses of the WebLogic
          > web
          > > servers that hold the session data for that request. If one goes down
          the
          > > other is perfectly capable of analyzing the cookie too. As long as one
          > proxy
          > > and one of your two clustered WL web servers survives your users will
          have
          > > intact sessions.
          > >
          > > You do, however, have a single point of failure at the app server level,
          > and
          > > at the database server level, compounded by the fact that both are on a
          > > single machine.
          > >
          > > Don't use WebLogic to run the cluster servlet. It's performance is
          > > terrible - we experienced a 6-7x performance degredation, and WL support
          > had
          > > no idea why. They wanted us to run a version of ClusterServlet with
          > timing
          > > code in it so that we could help them debug their code. I don't think
          so.
          > >
          > >
          > > Marko Milicevic <[email protected]> wrote in message
          > > news:[email protected]...
          > > > Our group is planning on configuring a two machine cluster to host
          > > > servlets/jsp's and a single backend app server to host all EJBs and a
          > > > database.
          > > >
          > > > IIS is going to be configured on each of the two cluster machines with
          a
          > > > cluster plugin. IIS is being used to optimize performance of static
          > HTTP
          > > > requests. All servlet/jsp request would be forwarded to the Weblogic
          > > > cluster. Resonate's Central Dispatch is also going to be installed on
          > the
          > > > two cluster machines. Central Dispatch is being used to provide HTTP
          > > > request load-balancing and to provide failover in case one of the IIS
          > > > servers fails (because the IIS process fails or the cluster machine
          it's
          > > on
          > > > fails).
          > > >
          > > > Will this configuration work? I'm most concerned about the failover
          of
          > > the
          > > > IIS cluster proxy. If one of the proxies is managing a sticky session
          > > (X),
          > > > what happens when the machine (the proxy is on) dies and we failover
          to
          > > the
          > > > other proxy? Is that proxy going to have any awareness of session X?
          > > > Probably not. The new proxy is probably going to believe this request
          > is
          > > > new and forward the request to a machine which may not host the
          existing
          > > > primary session. I believe this is an error?
          > > >
          > > > Is a cluster proxy a single-point-of-failure? Is there any way to
          avoid
          > > > this? Does the same problem exist if you use Weblogic's HTTP server
          (as
          > > the
          > > > cluster proxy)?
          > > >
          > > > Thank you.
          > > >
          > > > Marko.
          > > > .
          > > >
          > > >
          > > >
          > > >
          > > >
          > > >
          > >
          > >
          >
          >