Re: Throughput drops dramatically when I increase the number of instances in a cluster

Question

Thanks for the quick reply.
          Yes, we are running WLS with the performance pack.
          "Bernie Wong" <[email protected]> wrote in message
          news:[email protected]...
          > Fellow clusterers ....
          >
          > I have an EJB driver program that pumps work into a WLS cluster. For
          > scalability, I increase the number of drivers. The drivers have a big pool
          > of work to pump into the cluster.
          >
          > I noticed that with 6 (nothing special about that number) drivers, I can
          get
          > 500K worked items per hour in a single instance (clustered). That is, the
          > instance is cluster enabled and is watching for multicast, etc..
          >
          > When I add the second instance into the cluster, the cluster-wide
          throughput
          > with the six drivers drops to 370K items per hour. When I add the third
          > instance, the cluster-wide throughput drops further to 250K items per
          hour.
          > When I kill an instance, the throughput grows back to 370K items per hour.
          > When I kill another instance (to get back to one instance), the throughput
          > grows back to 500K/hr.
          >
          > Can somebody point out what we are doing wrong - do we need to tweek some
          > WLS parameters, do we have to avoid some features that limit cluster
          > scalability, etc..
          >
          > My WLS configuration is .....
          > - WLS 5.1, SP5
          > - JDK 1.2.1_04c with JIT, no Hotspot
          >
          > My hardware/software configuration consists of:
          > - an E4500 - 8 processor database server - this is only running at 15%
          > server busy
          > - three E4500 - all 8 processors/6GB RAM - application processors
          >
          > The cluster is implemented with one instance per application server.
          > Each application server is only 10-15% busy.
          >
          > My driver and application is ....
          > - stateless EJB driver and stateless EJB bean
          >
          > Your thoughts and comments would be greatly appreciated.
          >
          > Bernie
          >
          >
          

Answer 1

Thanks. It works. We screwed up the settings.
          "Bernie Wong" <[email protected]> wrote in message
          news:[email protected]...
          > Thanks for the reply.
          >
          > We ran the benchmark again and got the same scaling profile. This
          afternoon,
          > we are going to run a benchmark with a really simple driver + bean (a do
          > nothing bean) and see if we get the same scaling profile. From all the
          > responses, it would appear that WLS can scale therefore we want to isolate
          > the problem to either our code or our configuration (or both).
          >
          > I will let you know.
          >
          > Bernie
          >
          > "Rob Woollen" <[email protected]> wrote in message
          > news:[email protected]...
          > > Make sure that this was set on the client not the server.
          > >
          > > -- Rob
          > >
          > > Bernie Wong wrote:
          > > >
          > > > Jim,
          > > >
          > > > Thanks for the suggestions. One of my performance engineers ran
          several
          > > > experiments last week where he set the executeThreadCount to 6 and to
          10
          > > > along with a percentSocketReaders to 50. He saw no improvement. We
          will
          > > > rerun that experiment again since that seems to be the most common
          > > > recommendation.
          > > >
          > > > Bernie
          > > >
          > > > "Jim Zhou" <[email protected]> wrote in message
          > > > news:[email protected]...
          > > > > One possible reason is not enough socket reader execute threads, try
          > to do
          > > > > this:
          > > > > "-Dweblogic.system.executeThreadCount=10"
          > > > > "-Dweblogic.system.percentSocketReaders=50" on your client JVM
          startup
          > > > > command line.
          > > > >
          > > > > http://www.weblogic.com/docs51/admindocs/tuning.html#tuning clients
          > > > >
          > > > > WebLogic RMI
          > > > > If you are using WebLogic RMI clients and there are more than 2
          > Weblogic
          > > > > servers in a cluster, you may encounter a significant performance
          > > > > degradation (very long round trip times for stateless session beans,
          > for
          > > > > instance). The solution is to make some property changes on the
          client
          > > > side
          > > > > as explained below.
          > > > >
          > > > > The solution to this problem is to ensure that there are at least as
          > many
          > > > > socket reader threads as there are connections to the server and
          also
          > > > > allowing for some extra threads for processing other tasks. This is
          > > > > accomplished by starting the client with the command line argument
          > > > > "-Dweblogic.system.percentSocketReaders" set to a sufficiently high
          > > > > percentage (say 50) and by ensuring that there sufficient number of
          > > > execute
          > > > > threads for other processing on the client. A metric of twice the
          > number
          > > > of
          > > > > execute threads as there are servers in a cluster should work fine
          if
          > the
          > > > > above percentage is at 50. The command line argument affecting the
          > number
          > > > of
          > > > > execute threads is
          > > > >
          > > > > "-Dweblogic.system.executeThreadCount".
          > > > >
          > > > > For instance, we could use:
          > > > >
          > > > > "-Dweblogic.system.executeThreadCount=10"
          > > > >
          > > > > and
          > > > >
          > > > > "-Dweblogic.system.percentSocketReaders=50"
          > > > >
          > > > > when testing with 3 or 4 servers in a cluster.
          > > > >
          > > > >
          > > > > Bernie Wong <[email protected]> wrote in message
          > > > > news:[email protected]...
          > > > > > Thanks for the quick reply.
          > > > > >
          > > > > > Yes, we are running WLS with the performance pack.
          > > > > >
          > > > > >
          > > > > > "Bernie Wong" <[email protected]> wrote in message
          > > > > > news:[email protected]...
          > > > > > > Fellow clusterers ....
          > > > > > >
          > > > > > > I have an EJB driver program that pumps work into a WLS cluster.
          > For
          > > > > > > scalability, I increase the number of drivers. The drivers have
          a
          > big
          > > > > pool
          > > > > > > of work to pump into the cluster.
          > > > > > >
          > > > > > > I noticed that with 6 (nothing special about that number)
          drivers,
          > I
          > > > can
          > > > > > get
          > > > > > > 500K worked items per hour in a single instance (clustered).
          That
          > is,
          > > > > the
          > > > > > > instance is cluster enabled and is watching for multicast, etc..
          > > > > > >
          > > > > > > When I add the second instance into the cluster, the
          cluster-wide
          > > > > > throughput
          > > > > > > with the six drivers drops to 370K items per hour. When I add
          the
          > > > third
          > > > > > > instance, the cluster-wide throughput drops further to 250K
          items
          > per
          > > > > > hour.
          > > > > > > When I kill an instance, the throughput grows back to 370K items
          > per
          > > > > hour.
          > > > > > > When I kill another instance (to get back to one instance), the
          > > > > throughput
          > > > > > > grows back to 500K/hr.
          > > > > > >
          > > > > > > Can somebody point out what we are doing wrong - do we need to
          > tweek
          > > > > some
          > > > > > > WLS parameters, do we have to avoid some features that limit
          > cluster
          > > > > > > scalability, etc..
          > > > > > >
          > > > > > > My WLS configuration is .....
          > > > > > > - WLS 5.1, SP5
          > > > > > > - JDK 1.2.1_04c with JIT, no Hotspot
          > > > > > >
          > > > > > > My hardware/software configuration consists of:
          > > > > > > - an E4500 - 8 processor database server - this is only
          > running at
          > > > > 15%
          > > > > > > server busy
          > > > > > > - three E4500 - all 8 processors/6GB RAM - application
          > processors
          > > > > > >
          > > > > > > The cluster is implemented with one instance per application
          > > > server.
          > > > > > > Each application server is only 10-15% busy.
          > > > > > >
          > > > > > > My driver and application is ....
          > > > > > > - stateless EJB driver and stateless EJB bean
          > > > > > >
          > > > > > > Your thoughts and comments would be greatly appreciated.
          > > > > > >
          > > > > > > Bernie
          > > > > > >
          > > > > > >
          > > > > >
          > > > > >
          > > > >
          > > > >
          >
          >