Distributed queue - uneven load-balancing

I've read relevant threads, JMS : loadbalancing of messages is not happening uniformly between all the is close but no help.
I have 6 managed nodes in a cluster, nodes 1-3 on box A, nodes 4-6 on box B. I have a distributed queue + connection factory with server affinity disabled and load balancing enabled. Both the queue and connection factory are deployed (targeted) to the entire cluster. The sender on node 1 pushed 100k messages into the queue and 33k ended up evenly distributed (right down to a message!) across physical queues on 3 nodes on box A. Queues on nodes 4-6 received no messages. During the test all physical queues were alive and well, producers and consumers (MDBs) were alive and well, all nodes were up and so on. In the cluster config I said I've got 6 nodes (Number Of Servers In Cluster Address field). Please tell me something is wrong with my config and this is not a bug that I have to bring up with Support..
I am on 10.3.3 on Linux 64-bit wit JRockit 1.6

Hi,
No, this is not the correct way the JNDI tree supposed to look for distributed queue in a single cluster with two hosts. This means that either your configuration is wrong or both the hosts are not connected to each other properly.
In your scenario this should be your architecture
Box-A
=====
AS
MS-1, MS-2, MS-3 under Cluster
JMSServer-1 => MS-1
JMSServer-2 => MS-2
JMSServer-3 => MS-3
JMS_Module => Cluster
SubDeployment_UDQ => JMSServer-1, JMSServer-2, JMSServer-3
ConnFacty => Cluster
UDQ => SubDeployment_UDQ
Box-B
======
MS-4, MS-5, MS-6 under Cluster
JMSServer-4 => MS-4
JMSServer-5 => MS-5
JMSServer-6 => MS-6
JMS_Module => Cluster
SubDeployment_UDQ => JMSServer-4, JMSServer-5, JMSServer-6
ConnFacty => Cluster
UDQ => SubDeployment_UDQ
Where: *=>* means targeted to
In Box-A and Box-B the Cluster, SubDeployment_UDQ, ConnFacty and UDQ are the same NOT different.
Once this is done then you should be able to see UDQ and ConnFacty on all the servers JNDI tree then its the right configuration and now when you send messages on UDQ using the ConnFacty with "affinity disabled" you will see the messages getting distributed properly on all the 6 servers.
Hope this helps you.
Regards,
Ravish Mody

Similar Messages

  • Distributed services with load balancing and failover?

    Hullo;
    What platform would you use to implement something like the following:
    * easy registration of various services
    * delegation of a service request to the best candidate of many, based on some measure (probably reported by the services themselves)
    * quick failover and location of an alternate service in case the best candidate does not respond (real-life environment, uncertain networks and servers;)
    RMI could be a starting point, with a custom SocketFactory to take care of the timeouts and redelegations and a good delegator service to work through. The service concept sounds a lot like JINI, but I don't see any provision for best candidate selection, and wonder whether JINI would really save any time compared to RMI in this case.
    Is there anything else I should be aware of? I wouldn't mind finding a pre-built wheel. (Cougaar (http://www.cougaar.org/) is on my reading list; a quick glance gives me the impression it may be a bit too heavy on the communication level, but maybe I'm wrong.)
    Thanks for your thoughts;
    //ata

    ata,
    Before jumping to anything so bloated and limited as cougaar, take time to consider what you really need. Before grabing at the fanciest Java features like RMI, JINI, and custom SocketFactories; focus on what you are trying to accomplish.
    There are plenty of great answers right here, at this forum.
    Good hunting,
    John

  • Distributed Queue - Unable To Load Balance Between Each Time A Send Method Is Called

    Hi,
              According to the JMS documentation, I should be able to get the
              distributed queue to load balance
              between each time the message producer calls Message.send(). I was not
              able to achieve this, however,
              I noticed the load balancing happens when a JMS client is stopped and
              restarted (meaning totally
              exit the JVM and restart the JVM).
              Here is my configuration:
              WLS 8.1 SP2 on XP
              One cluster with two nodes (running on the same machine w/ different port)
              Each node hosts one JMS server, which hosts one physical queue and using
              JDBC store
              One distributed queue with two physical members from each of the JMS
              server.
              JMS Connection Factory is configured with "Load Balancing Enabled" set to
              yes,
              and "Server Affinity Enabled" to no. This connection factory is target to
              the cluster.
              The queue session for the queue sender is created with transaction setting
              to false.
              Any hints and ideas would greatly appreciated.
              Here is the content of config.xml:
              ========================================================================
              <?xml version="1.0" encoding="UTF-8"?>
              <Domain ConfigurationVersion="8.1.0.0" Name="odh">
              <Cluster ClusterAddress="localhost:8001,localhost:9001"
              MulticastAddress="237.0.0.1" Name="odhCluster_1"/>
              <Server ListenAddress="" ListenPort="7001" Machine="localhost"
              Name="odhAdmin" NativeIOEnabled="true" ServerVersion="8.1.2.0">
              <SSL Enabled="false" HostnameVerificationIgnored="false"
              IdentityAndTrustLocations="KeyStores" Name="odhAdmin"/>
              </Server>
              <Server Cluster="odhCluster_1" ExpectedToRun="false"
              IIOPEnabled="false" ListenAddress="" ListenPort="8001"
              Machine="localhost" Name="odhManagedServer_1"
              NativeIOEnabled="true" ServerVersion="8.1.2.0">
              <SSL Enabled="false" IdentityAndTrustLocations="KeyStores"
              Name="odhManagedServer_1"/>
              <ExecuteQueue Name="weblogic.kernel.Default" ThreadCount="15"/>
              </Server>
              <Server Cluster="odhCluster_1" ExpectedToRun="false"
              IIOPEnabled="false" ListenAddress="" ListenPort="9001"
              Machine="localhost" Name="odhManagedServer_2"
              NativeIOEnabled="true" ServerVersion="8.1.2.0">
              <SSL Enabled="false" IdentityAndTrustLocations="KeyStores"
              Name="odhManagedServer_2"/>
              <ExecuteQueue Name="weblogic.kernel.Default" ThreadCount="15"/>
              </Server>
              <MigratableTarget Cluster="odhCluster_1"
              Name="odhManagedServer_1 (migratable)"
              Notes="This is a system generated default migratable target for a
              server. Do not delete manually."
              UserPreferredServer="odhManagedServer_1"/>
              <MigratableTarget Cluster="odhCluster_1"
              Name="odhManagedServer_2 (migratable)"
              Notes="This is a system generated default migratable target for a
              server. Do not delete manually."
              UserPreferredServer="odhManagedServer_2"/>
              <Machine Name="localhost">
              <NodeManager ListenAddress="localhost" Name="localhost"/>
              </Machine>
              <JMSConnectionFactory AcknowledgePolicy="All"
              DefaultDeliveryMode="Persistent"
              JNDIName="com.neoforma.ConnectionFactory"
              Name="odhConnectionFactory" ServerAffinityEnabled="false"
              Targets="odhCluster_1" XAConnectionFactoryEnabled="true"/>
              <JMSDistributedQueue JNDIName="com.neoforma.odhDistributedQueue_1"
              LoadBalancingPolicy="Round-Robin" Name="odhDistributedQueue_1"
              Targets="odhCluster_1">
              <JMSDistributedQueueMember JMSQueue="odhQueue_1"
              Name="DistributedQueueMember_1"/>
              <JMSDistributedQueueMember JMSQueue="odhQueue_2"
              Name="DistributedQueueMember_2"/>
              </JMSDistributedQueue>
              <JMSJDBCStore ConnectionPool="odhMessagePool"
              Name="odhJMSJDBCStore_1" PrefixName="Order1_"/>
              <JMSJDBCStore ConnectionPool="odhMessagePool"
              Name="odhJMSJDBCStore_2" PrefixName="Order2_"/>
              <JMSServer Name="odhJMSServer_1" Store="odhJMSJDBCStore_1"
              Targets="odhManagedServer_1">
              <JMSQueue CreationTime="1076439896999"
              JNDIName="com.neoforma.odhQueue_1" Name="odhQueue_1"
              StoreEnabled="true"/>
              </JMSServer>
              <JMSServer Name="odhJMSServer_2" Store="odhJMSJDBCStore_2"
              Targets="odhManagedServer_2">
              <JMSQueue CreationTime="1076439664343"
              JNDIName="com.neoforma.odhQueue_2" Name="odhQueue_2"
              StoreEnabled="true"/>
              </JMSServer>
              <JDBCConnectionPool
              DriverName="oracle.jdbc.xa.client.OracleXADataSource"
              Name="odhConnectionPool" Password="...."
              Properties="user=..." Targets="odhCluster_1"
              TestTableName="SQL SELECT 1 FROM DUAL" URL="................."/>
              <JDBCConnectionPool DriverName="oracle.jdbc.driver.OracleDriver"
              Name="odhMessagePool" Password="....."
              Properties="user=....." Targets="odhCluster_1"
              TestTableName="SQL SELECT 1 FROM DUAL" URL="............."/>
              <JDBCMultiPool AlgorithmType="High-Availability"
              Name="odhJDBCMultiPool_1"
              PoolList="odhConnectionPool,odhMessagePool"
              Targets="odhCluster_1"/>
              <JDBCTxDataSource EnableTwoPhaseCommit="false"
              JNDIName="com.neoforma.order.orderDS" Name="odhJDBCDataSource_1"
              PoolName="odhConnectionPool" Targets="odhCluster_1"/>
              <Security Name="odh" PasswordPolicy="wl_default_password_policy"
              Realm="wl_default_realm" RealmSetup="true"/>
              <!--
              <EmbeddedLDAP
              Credential="{3DES}j+xkS9y1EYJUfic+M9ZJ+5DqGjiwTaVnt+Ti0TQWxXg="
              Name="odh"/>
              <SecurityConfiguration
              Credential="{3DES}OiyDMEOJS4gPLumKeKYWC+Kj9xWib6MhbmrNjeBmjJ7bpJypNb6Z7bUtAQF/bvi2RrFMs+3kqKerWNyD3NyT3QsrsyPoBDT0"
              Name="odh" RealmBootStrapVersion="1"/>
              -->
              <Realm FileRealm="wl_default_file_realm" Name="wl_default_realm"/>
              <FileRealm Name="wl_default_file_realm"/>
              <PasswordPolicy Name="wl_default_password_policy"/>
              <Application Deployed="true" Name="odh.ear"
              Path="D:\bea\user_projects\domains\odh\applications\odh.ear"
              StagedTargets="odhManagedServer_1,odhManagedServer_2"
              StagingMode="stage" TwoPhase="true">
              <EJBComponent Name="odh.jar" Targets="odhCluster_1" URI="odh.jar"/>
              </Application>
              <StartupClass ClassName="com.neoforma.startup.JMXMBeanStartup"
              DeploymentOrder="1" Name="ODH MBean Startup Class"
              Notes="ODH MBean Startup Class - Note" Targets="odhAdmin"/>
              <EmbeddedLDAP
              Credential="{3DES}YFY55/dsdxI9HL/AKGRXHuR1VwyJewNFdAHdrtk/WMM="
              Name="odh"/>
              <SecurityConfiguration
              Credential="{3DES}ZCPa1Bsrj3z2DhVKVUbq32zTYipDVff+LDB9+1b2Dr4VLhz5yjZyHgPheqS/kum4VVZamDYN07Hyb6rALiCTHhwt1EzK5+M+"
              Name="odh" RealmBootStrapVersion="1"/>
              </Domain>
              

    Thanks for the Makiey. I am surprise that BEA hasn't come back with any
              info.
              Hien
              On 7 Jul 2004 01:51:01 -0700, makiey <[email protected]> wrote:
              >
              > Hi Hien Luu,
              >
              > We also have a problem with load balancing, tested with WLS 7.0 SP4 and
              > WLS 8.1
              > SP2 (HP UX). The only "working" configuration is load-balancing policy =
              > random
              > (CF deployed to cluster, load balancing enabled, affinity disabled).
              > With the
              > "round-robin" policy we cannot utilize more than 50% dis. queue's
              > members.
              >
              > I'm trying to prepare a reproducer...
              >
              > greetings,
              > makiey
              >
              >
              > "Hien Luu" <[email protected]> wrote:
              >> Hi,
              >>
              >> According to the JMS documentation, I should be able to get the =
              >>
              >> distributed queue to load balance
              >> between each time the message producer calls Message.send(). I was not
              >> =
              >> =
              >>
              >> able to achieve this, however,
              >> I noticed the load balancing happens when a JMS client is stopped and
              >> =
              >>
              >> restarted (meaning totally
              >> exit the JVM and restart the JVM).
              >>
              >> Here is my configuration:
              >>
              >> WLS 8.1 SP2 on XP
              >> One cluster with two nodes (running on the same machine w/ different
              >> por=
              >> t)
              >> Each node hosts one JMS server, which hosts one physical queue and
              >> using=
              >> =
              >>
              >> JDBC store
              >> One distributed queue with two physical members from each of the JMS
              >> =
              >>
              >> server.
              >> JMS Connection Factory is configured with "Load Balancing Enabled" set
              >> t=
              >> o =
              >>
              >> yes,
              >> and "Server Affinity Enabled" to no. This connection factory is target
              >> =
              >> to =
              >>
              >> the cluster.
              >>
              >> The queue session for the queue sender is created with transaction
              >> setti=
              >> ng =
              >>
              >> to false.
              >>
              >> Any hints and ideas would greatly appreciated.
              >>
              >>
              >> Here is the content of config.xml:
              >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
              >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
              >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
              >>
              >> <?xml version=3D"1.0" encoding=3D"UTF-8"?>
              >> <Domain ConfigurationVersion=3D"8.1.0.0" Name=3D"odh">
              >> <Cluster ClusterAddress=3D"localhost:8001,localhost:9001"
              >> MulticastAddress=3D"237.0.0.1" Name=3D"odhCluster_1"/>
              >> <Server ListenAddress=3D"" ListenPort=3D"7001" Machine=3D"localhost=
              >> "
              >> Name=3D"odhAdmin" NativeIOEnabled=3D"true" ServerVersion=3D"8.1=
              >> .2.0">
              >> <SSL Enabled=3D"false" HostnameVerificationIgnored=3D"false"
              >> IdentityAndTrustLocations=3D"KeyStores" Name=3D"odhAdmin"/>=
              >>
              >> </Server>
              >> <Server Cluster=3D"odhCluster_1" ExpectedToRun=3D"false"
              >> IIOPEnabled=3D"false" ListenAddress=3D"" ListenPort=3D"8001"
              >> Machine=3D"localhost" Name=3D"odhManagedServer_1"
              >> NativeIOEnabled=3D"true" ServerVersion=3D"8.1.2.0">
              >> <SSL Enabled=3D"false" IdentityAndTrustLocations=3D"KeyStores"
              >> =
              >> =
              >>
              >> Name=3D"odhManagedServer_1"/>
              >> <ExecuteQueue Name=3D"weblogic.kernel.Default" ThreadCount=3D"1=
              >> 5"/>
              >> </Server>
              >> <Server Cluster=3D"odhCluster_1" ExpectedToRun=3D"false"
              >> IIOPEnabled=3D"false" ListenAddress=3D"" ListenPort=3D"9001"
              >> Machine=3D"localhost" Name=3D"odhManagedServer_2"
              >> NativeIOEnabled=3D"true" ServerVersion=3D"8.1.2.0">
              >> <SSL Enabled=3D"false" IdentityAndTrustLocations=3D"KeyStores"
              >> =
              >> =
              >>
              >> Name=3D"odhManagedServer_2"/>
              >> <ExecuteQueue Name=3D"weblogic.kernel.Default" ThreadCount=3D"1=
              >> 5"/>
              >> </Server>
              >> <MigratableTarget Cluster=3D"odhCluster_1"
              >> Name=3D"odhManagedServer_1 (migratable)"
              >> Notes=3D"This is a system generated default migratable target
              >> f=
              >> or a =
              >>
              >> server. Do not delete manually."
              >> UserPreferredServer=3D"odhManagedServer_1"/>
              >> <MigratableTarget Cluster=3D"odhCluster_1"
              >> Name=3D"odhManagedServer_2 (migratable)"
              >> Notes=3D"This is a system generated default migratable target
              >> f=
              >> or a =
              >>
              >> server. Do not delete manually."
              >> UserPreferredServer=3D"odhManagedServer_2"/>
              >> <Machine Name=3D"localhost">
              >> <NodeManager ListenAddress=3D"localhost" Name=3D"localhost"/>
              >> </Machine>
              >> <JMSConnectionFactory AcknowledgePolicy=3D"All"
              >> DefaultDeliveryMode=3D"Persistent"
              >> JNDIName=3D"com.neoforma.ConnectionFactory"
              >> Name=3D"odhConnectionFactory" ServerAffinityEnabled=3D"false"
              >> Targets=3D"odhCluster_1" XAConnectionFactoryEnabled=3D"true"/>
              >> <JMSDistributedQueue JNDIName=3D"com.neoforma.odhDistributedQueue_1=
              >> "
              >> LoadBalancingPolicy=3D"Round-Robin" Name=3D"odhDistributedQueue=
              >> _1" =
              >>
              >> Targets=3D"odhCluster_1">
              >> <JMSDistributedQueueMember JMSQueue=3D"odhQueue_1" =
              >>
              >> Name=3D"DistributedQueueMember_1"/>
              >> <JMSDistributedQueueMember JMSQueue=3D"odhQueue_2" =
              >>
              >> Name=3D"DistributedQueueMember_2"/>
              >> </JMSDistributedQueue>
              >> <JMSJDBCStore ConnectionPool=3D"odhMessagePool"
              >> Name=3D"odhJMSJDBCStore_1" PrefixName=3D"Order1_"/>
              >> <JMSJDBCStore ConnectionPool=3D"odhMessagePool"
              >> Name=3D"odhJMSJDBCStore_2" PrefixName=3D"Order2_"/>
              >> <JMSServer Name=3D"odhJMSServer_1" Store=3D"odhJMSJDBCStore_1"
              >> =
              >>
              >> Targets=3D"odhManagedServer_1">
              >> <JMSQueue CreationTime=3D"1076439896999"
              >> JNDIName=3D"com.neoforma.odhQueue_1" Name=3D"odhQueue_1"
              >> =
              >>
              >> StoreEnabled=3D"true"/>
              >> </JMSServer>
              >> <JMSServer Name=3D"odhJMSServer_2" Store=3D"odhJMSJDBCStore_2"
              >> =
              >>
              >> Targets=3D"odhManagedServer_2">
              >> <JMSQueue CreationTime=3D"1076439664343"
              >> JNDIName=3D"com.neoforma.odhQueue_2" Name=3D"odhQueue_2"
              >> =
              >>
              >> StoreEnabled=3D"true"/>
              >> </JMSServer>
              >> <JDBCConnectionPool
              >> DriverName=3D"oracle.jdbc.xa.client.OracleXADataSource"
              >> Name=3D"odhConnectionPool" Password=3D"...."
              >> Properties=3D"user=3D..." Targets=3D"odhCluster_1"
              >> TestTableName=3D"SQL SELECT 1 FROM DUAL" URL=3D"...............=
              >> .."/>
              >> <JDBCConnectionPool DriverName=3D"oracle.jdbc.driver.OracleDriver"
              >> Name=3D"odhMessagePool" Password=3D"....."
              >> Properties=3D"user=3D....." Targets=3D"odhCluster_1"
              >> TestTableName=3D"SQL SELECT 1 FROM DUAL" URL=3D"............."/=
              >>>
              >> <JDBCMultiPool AlgorithmType=3D"High-Availability"
              >> Name=3D"odhJDBCMultiPool_1"
              >> PoolList=3D"odhConnectionPool,odhMessagePool" =
              >>
              >> Targets=3D"odhCluster_1"/>
              >> <JDBCTxDataSource EnableTwoPhaseCommit=3D"false"
              >> JNDIName=3D"com.neoforma.order.orderDS" Name=3D"odhJDBCDataSour=
              >> ce_1"
              >> PoolName=3D"odhConnectionPool" Targets=3D"odhCluster_1"/>
              >> <Security Name=3D"odh" PasswordPolicy=3D"wl_default_password_policy=
              >> "
              >> Realm=3D"wl_default_realm" RealmSetup=3D"true"/>
              >> <!--
              >> <EmbeddedLDAP
              >> Credential=3D"{3DES}j+xkS9y1EYJUfic+M9ZJ+5DqGjiwTaVnt+Ti0TQWxXg=
              >> =3D" =
              >>
              >> Name=3D"odh"/>
              >> <SecurityConfiguration
              >> Credential=3D"{3DES}OiyDMEOJS4gPLumKeKYWC+Kj9xWib6MhbmrNjeBmjJ7=
              >> bpJypNb6Z7bUtAQF/bvi2RrFMs+3kqKerWNyD3NyT3QsrsyPoBDT0"
              >> Name=3D"odh" RealmBootStrapVersion=3D"1"/>
              >> -->
              >> <Realm FileRealm=3D"wl_default_file_realm" Name=3D"wl_default_realm=
              >> "/>
              >> <FileRealm Name=3D"wl_default_file_realm"/>
              >> <PasswordPolicy Name=3D"wl_default_password_policy"/>
              >> <Application Deployed=3D"true" Name=3D"odh.ear"
              >> Path=3D"D:\bea\user_projects\domains\odh\applications\odh.ear"
              >> StagedTargets=3D"odhManagedServer_1,odhManagedServer_2"
              >> StagingMode=3D"stage" TwoPhase=3D"true">
              >> <EJBComponent Name=3D"odh.jar" Targets=3D"odhCluster_1" URI=3D"=
              >> odh.jar"/>
              >> </Application>
              >> <StartupClass ClassName=3D"com.neoforma.startup.JMXMBeanStartup"
              >> DeploymentOrder=3D"1" Name=3D"ODH MBean Startup Class"
              >> Notes=3D"ODH MBean Startup Class - Note" Targets=3D"odhAdmin"/>=
              >>
              >> <EmbeddedLDAP
              >> Credential=3D"{3DES}YFY55/dsdxI9HL/AKGRXHuR1VwyJewNFdAHdrtk/WMM=
              >> =3D" =
              >>
              >> Name=3D"odh"/>
              >> <SecurityConfiguration
              >> Credential=3D"{3DES}ZCPa1Bsrj3z2DhVKVUbq32zTYipDVff+LDB9+1b2Dr4=
              >> VLhz5yjZyHgPheqS/kum4VVZamDYN07Hyb6rALiCTHhwt1EzK5+M+"
              >> Name=3D"odh" RealmBootStrapVersion=3D"1"/>
              >> </Domain>
              >
              Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
              

  • How to process one message at a time on a distributed queue

    I have a situation where I need to execute/process a task of a type
              asynchronously and this should be done one at a time. To do that I am thinking of setting
              up a distributed queue (for load balancing and failover). Now I have
              this requirement that only one task should be processed at one time. So
              for example if two messages (m1 followed by m2) are sent to distributed
              queue and I have the distributed queue which points to two physical
              queues on two different instances of weblogic(wl1 and wl2), the
              distributed queue will likely send m1 to wl1 and m2 to wl2 (assuming
              distributed queue is setup with round robin policy). If m1 has started
              executing/processing in my MDB on wl1, I do not want start processing
              m2 on wl2 untill m1 has been processed by wl1. Is it possible to do
              this on weblogic 8.1 JMS?
              If this is not supported I was thinking of implementing a database
              based locking framework and each mdb will use that framework to check
              if a task of a particular type is being processed before processing a
              new task. Also in this case I will need to use max. 1 mdb for each
              queue on each weblogic server.
              --Navjeet                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

    Hello navjeet,
              I don't think there are any parameters that you can set on a distributed queue to ensure messages are handled in the sequential manner you describe so it looks like you will have to implement your own locking mechanism.
              In reposponse to your other question you can set the number of MDbeans in the free pool to 1 (or the number of queues you have) in the MDBean deployment descriptor.
              see:
              http://e-docs.bea.com/wls/docs81/ejb/message_beans.html#1120592
              Cheers,
              Hoos

  • CSS 11503 Load Balancing Verification

    Alright, so I have toiled long and hard to get this right.  I think I have the config down but I am unsure on how to verify how this load balancing is working.
    Here is the Content Config that I am speaking of:
    content cad-rule
        add service wls1-e0
        add service wls1-e1
        add service wls2-e0
        add service wls2-e1
        add service wls3-e0
        add service wls3-e1
        add service wls4-e0
        add service wls4-e1
        add service wls5-e0
        add service wls5-e1
        add service wls6-e0
        add service wls6-e1
        arrowpoint-cookie expiration 00:00:15:00
        advanced-balance arrowpoint-cookie
        redundant-index 2
        vip address 172.30.194.195 range 2
        arrowpoint-cookie name TOQ
        protocol tcp
        port 8001
        url "/*"
        active
    Each service in the rule above is configured as follows:
    service wls1-e1
      port 8001
      protocol tcp
      strin ags001-e1
      ip address 172.30.193.81
      keepalive type http
      keepalive uri "/cad/index.html"
      redundant-index 12
      keepalive frequency 20
      keepalive maxfailure 10
      keepalive retryperiod 2
      active
    I am using the advanced arrowpoint cookies because I need some stickiness here.  Straight round-robin would not have done what I needed it to do.
    Now, when I go to my show summary, this is what I see for this rule:
                     cad-rule    Master   wls1-e0 84274
                                                wls1-e1 13144
                                                wls2-e0 96884
                                                wls2-e1 26374
                                                wls3-e0 71145
                                                wls3-e1 16592
                                                wls4-e0 76403
                                                wls4-e1 8657
                                                wls5-e0 118623
                                                wls5-e1 22760
                                                wls6-e0 30836
                                                wls6-e1 20464
    The far right column indicates the services hits.  I originally had the E1's suspended and activated them later on. So if this was true round robin, all the E0's should have the same number of service hits and all the E1's should have the same number of service hits.  But as you can see, the wls5 server is getting hit the most while the wls6 server is sitting there twiddling its thumbs.
    Now understanding how the arrowpoint cookies do their load balancing (inserting a cooking into the flow and then timing out after 15 mins as configured above) I would not expect a 1:1 ratio of load balancing between servers.  But the distribution above seems rather extreme.
    Does anyone have any suggestions on how to both A) verify that this is the right config and B) suggest to my boss that this is working the way it should be working?
    Thanks!
    James

    Hi James,
    There are several reasons of the uneven load balancing that you are seeing (based on the show summary). First
    of all, the CSS is configured to do stickiness (advance-balance).
    With arrowpoint-cookies (for HTTP only) method for stickiness, only the requests coming with the same cookie
    are going to get stuck to the same server, since the cookie is
    lost when the browser is closed (or based on the expiration), then the stickiness is going to be session
    based and if the same client open a new session is going to be load balanced.
    Is important to understand that when using stickiness, no real even load balancing is
    going to happen since we are sticking new flows to the same server; even when layer 5 stickiness would
    permit more even balancing than layer 3 stickiness (source IP based).
    Also consider that the "show summary" is a command to see the hits (requests) being balanced to an specific
    server, this is a good command to see the load balancing, anyway since the CSS balance
    connections (flows), a persistent connection could have a lot of requests, so all those requests are
    always going to the same server (incrementing the amount of hits in the counter) while a non-persistent
    connection would be just one request (refer to HTTP persistence).
    Also keep in mind that if a service is take out for maintenance, or is added to the load balancing later
    than another, or if goes down for a period of time, then the CSS will be balancing among the remaining alive
    servers. When you add the server again, the another servers are going to have connections
    already established, so since the CSS is doing round robin, the server last added will
    never have the same amount of connections (nor hits) that the other ones, because while one could
    have 55 for example, the new one will have it first connection, and when the first one
    gets the 56, the another will get the second, and so on.
    Please let me know if this makes any sense.
    Diego M

  • Load Balancing Two Different Waas Models

    Is it possible to deploy two waas devices having different models? Specifically WAVE-694-K9 and
    WAVE-7541-K9? Thanks

    Hi,
    I guess you questions is about having different WAAS models part of the same WCCP farm.
    If that is the case, it shouldn't be an issue at all.
    The only problem I would see with this is that if you have even load balancing, the less powerful device will be overloaded while there is still capacity on the other device
    If you run into this, you might try to play with the WCCP mask to achieve uneven load balancing based on the devices processing powers.
    Regards,
    Nicolas

  • OSB 10gR3 (WLS 10.3) - Distributed Queues & Load Balancing

    I have a question in relation to distributed queues and its JMS proxy service consumer in OSB
    I've set up a uniform distributed queue deployed using a sub-deployment resulting in the queue being targeted to the respective JMS servers in the cluster.
    I've then set up a messaging service using JMS as the transport with the following URI
    jms://server1:7011,server2:7012/weblogic.jms.XAConnectionFactory/myQueue
    When I look at the monitoring tab of my distributed queue, I can see 16 current consumers to one of the members but none for the other one. My understanding is that the proxy is just a mere MDB and as such I thought WLS was optimised to make sure all MDB instances would listen to all members of the distributed queue. Why do I have 16 consumers to one member only?
    Since only one member has consumers, any producer will always push messages to this member only. (I believe it is optimised to get a member with consumer(s) if any available)
    I've also tried to use a custom Connection Factory deployed the same way my distributed queue was, and ensure the connection factory had load balancing enabled. But no success with this either.
    jms://server1:7011,server2:7012/jms.MyConnectionFactory/myQueue
    I looked at the deployment - though not directly performed by me but rather the bus console - and it looks like the application is targeted to the cluster.
    How can I achieve true load balancing here, ensuring both members are consumed by my JMS proxy service?
    In that case, would any produced message go to either member then as both have consumers?
    Also, is the load balancing decision made by the producer when the Queue connection is created?
    If so, how do you achieve true load balancing? Do you need to ask for a new Q connection each time you want to send a message rather than caching the connection?
    Hope I am clear enough
    Thanks
    Arnaud

    This confused me too!
    The way I understand it, is that, as you say, a proxy service is like a single MDB. The MDB will bind to the queue it first finds when it connects.
    The URL that you specify which contains your two servers but the first address in the URL is the one that will be used for the connection. If the first server is unavailable, then the second one will be used.
    If you have a distributed queue, this doesn’t help much, as you do end up with one of the queue members with no consumers on it.
    You can configure a forward delay for the distributed queue, which will cause WLS to forward messages to a queue with consumers, but this isn’t a good idea if you have large JMS messages as WLS needs to serialize and de-serialize across the network to move the message.
    I think that what you have to do, is define two proxy services, one connecting to the first server, and the other connecting to the second.
    I haven’t found a better way so far, but it does seem a bit over the top, but then, if you wrote a an external java client which attached to a distributed queue you would specify the connection url and it would behave in the same way – if you wanted it to bind to both distributed destination members, you would have to code it or run two instances, so maybe its just working as it should – even though it seems strange.
    I think the producer will simply load balance across the distributed queue members, it doesn’t pay any attention whether there are to consumers attached – this happened to me the other day!!
    Pete

  • JMS cluster and distributed destination load balancing question

              Hi All
              Scenario: 2 WL 7 servers in cluster with distributed queue in both of them and
              both the servers have an MDB deployed for the queue. Now if a producer in server
              #1 writes to the Queue - he will write to the local queue - right?
              In that case will the local MDB pick up the message or that can be load balanced?
              OR the write it self can be load balanced?
              I really want either the write or the read to be load balanced - but I suspect
              server affinity will play a mess here. Can anyone pls clarify.
              thanks
              Anamitra
              

              Hi All
              Scenario: 2 WL 7 servers in cluster with distributed queue in both of them and
              both the servers have an MDB deployed for the queue. Now if a producer in server
              #1 writes to the Queue - he will write to the local queue - right?
              In that case will the local MDB pick up the message or that can be load balanced?
              OR the write it self can be load balanced?
              I really want either the write or the read to be load balanced - but I suspect
              server affinity will play a mess here. Can anyone pls clarify.
              thanks
              Anamitra
              

  • Getting load-balancing with distributed destination to work...

    Hello,
              I try to setup a proof of concept for balancing heavy load over several JMS server instances on WLS 8.1 SP4:
              I have
              - 2 managed servers in 1 cluster plus adminserver on 1 machine (WinXP)
              - 1 JMS server on each server (no migrateable targets used), having 1 physical queue
              - 1 distributed queue deployed to the cluster, consisting of the two physical queues
              - 1 connection factory deployed to the cluster with round-robin load-balancing enabled, and server affinity disabled
              - 1 test JSP using the connection factory from above, doing a complete re-connection per test message
              - 1 MDB ejb module deployed to the cluster
              and the result is:
              1. Calling the JSP through the second server instance load-balances messages on both JMS servers, that's fine ...
              BUT
              2. Calling the JSP through the first server instance processes all messages on the first server instance's JMS server, no message is ever sent to the second server.
              What could be the reason for the different behaviour of both servers?
              extract from config.xml used:
              <Cluster
              MulticastAddress="237.0.0.1"
              Name="clusterA"/>
              <EJBComponent Name="jtest"
              Targets="clusterA"
              URI="ejb/test.jar"/>
              <WebAppComponentName="test"
              Targets="clusterA"
              URI="web/test"/>
              <JMSConnectionFactory
              JNDIName="ConnectionFactory"
              LoadBalancingEnabled="true"
              ServerAffinityEnabled="false"
              Targets="clusterA"/>
              <JMSServer
              Name="JMSServer1"
              Store="JMSFileStore1"
              Targets="server1">
              <JMSQueue
              JNDIName="TestQueue.server1"
              JNDINameReplicated="true"
              Name="TestQueue-server1"/>
              </JMSServer>
              <JMSServer
              Name="JMSServer2"
              Store="JMSFileStore3"
              Targets="server2">
              <JMSQueue
              JNDIName="TestQueue.server2"
              JNDINameReplicated="true"
              Name="TestQueue-server2"/>
              </JMSServer>
              <JMSDistributedQueue
              JNDIName="TestQueue.DD"
              Name="TestQueueDD"
              Targets="clusterA">
              <JMSDistributedQueueMember
              JMSQueue="TestQueue.server1"
              Name="TestQueue.server1Memeber"/>
              <JMSDistributedQueueMember
              JMSQueue="TestQueue.server2"
              Name="TestQueue.server2Member"/>
              </JMSDistributedQueue>
              Cheers
              Martin

    Ok solved, it works as expected, when running on two solaris hosts

  • Load Balancing vs Load Distributed

    Hi,
    What is difference of these?

    Hi,
    It seems that you have posted this question in the forum before:
    https://social.technet.microsoft.com/Forums/en-US/4db5f816-880e-4f0c-b361-2f245e080014/nlb-vs-load-distributed?forum=winserverClustering
    "From the load distribution definition in Exchange service, the load distribution is in distributing the load evenly when all servers are available. However, when one or more servers are unavailable, the load distribution may become uneven among
    the remaining servers, especially if your organization is distributed across multiple Active Directory sites.
    Network Load Balancing (NLB) is one of the load distribution technologies. Network Load Balancing has many advantages over other load balancing solutions that can introduce single points of failure or performance bottlenecks. Because there are no special
    hardware requirements for Network Load Balancing, you can use any industry standard compatible computer in a Network Load Balancing cluster.
    What Is Network Load Balancing
    http://technet.microsoft.com/en-us/library/cc779570(v=WS.10).aspx"
    Since you had marked the reply above as an answer, do you have any other details you want to know? In addition, if the answer above is not helpful, I would appreciate it if you can explain the meaning of "Load Distributed".
    Best regards,
    Susie

  • Load balancing and clustered MDBs using a Queue

    With the following advice/information from
              http://e-docs.bea.com/wls/docs61/faq/jms.html#252978,
              "One customer had an example where topic MDBs are needed in which there will
              be multiple implementations of the MDBs listening on the same topic. In
              other words, more than one MDB with a different implementation may be
              subscribing to the same topic. The client has no advanced way of knowing how
              many different kinds of MDBs may be listening on the same topic, but it is
              possible for there to be more than one listener, therefore topics, not
              queues. For each kind of MDB listening on the topic, the message is
              delivered exactly once (i.e., the message will be delivered exactly once to
              an instance in each named MDB pool listening on the topic)."
              If I have a set of MDB's scraping Messages from the JMS Queue, what is the
              load-balancing behavior?
              1) Consider a lightly-loaded system, where each MDB has sufficient time to
              process the message before the next one shows up in the queue. Will the same
              MDB get all the messages?
              2) Consider a heavily-loaded system, where each MDB has insufficient time to
              process the message before the next one shows up in the queue. How would you
              characterize the de facto load-balancing behavior? least-loaded?
              

    Have you tried the SAP WebDispatcher.  This should acomplish what you want. 
    http://help.sap.com/saphelp_nw04/helpdata/en/42/5cfd3b0e59774ee10000000a114084/frameset.htm

  • 3rd party distributed SW load balancing with In-Memory Replication

              Hi,
              Could someone please comment on the feasibility of the following setup?
              I've started testing replication with a software load balancing product. This
              product lets all nodes receive all packets and uses a kernel-level filter
              to let only one node at the time receive it. Since there's minimum 1 heartbeat
              between the nodes, there are several NICs in each node.
              At the moment it seems like it doesn't work: - I use the SessionServlet - with
              a 2-node cluster I first have the 2 nodes up and I access it with a single client:
              .the LB is configured to be sticky wrt. source IP address, so the same node gets
              all the traffic - when I stop the node receiving the traffic the other node takes
              over (I changed the colours of SessionServlet) . however, the counter restarts
              at zero
              From what I read of the in-memory replication documentation I thought that it
              might work also with a distributed software load balancing cluster. Any comments
              on the feasability of this?
              Is there a way to debug replication (in WLS6SP1)? I don't see any replication
              messages in the logs, so I'm not even sure that it works at all. - I do get a
              message about "Clustering Services startting" when I start the examples server
              on each node - is there anything tto look for in the console to make sure that
              things are working? - the evaluation license for WLS6SP1 on NT seems to support
              In-Memory Replication and Cluster. However, I've also seen a Cluster-II somewhere:
              is that needed?
              Thanks for your attention!
              Regards, Frank Olsen
              

    We are considering Resonate as one of the software load balancer. We haven't certified
              them yet. I have no idea how long its going to take.
              As a base rule if the SWLB can do the load balancing and maintain stickyness that is fine
              with us as long as it doesn't modify the cookie or the URL if URL rewriting is enabled.
              Having said that if you run into problems we won't be able to support you since it is not
              certified.
              -- Prasad
              Frank Olsen wrote:
              > Prasad Peddada <[email protected]> wrote:
              > >Frank Olsen wrote:
              > >
              > >> Hi,
              > >>
              > > We don't support any 3rd party software load balancers.
              >
              > Does that mean that there are technical reasones why it won't work, or just that
              > you haven't tested it?
              >
              > > As >I said before I am thinking your configuration is >incorrect if n-memory
              > replication is not working. I would >strongly suggest you look at webapp deployment
              > descriptor and >then the config.xml file.
              >
              > OK.
              >
              > >Also doing sticky based on source ip address is not good. You >should do it based
              > on passive cookie persistence or active >cookie persistence (with cookie insert,
              > a new one).
              > >
              >
              > I agree that various source-based sticky options (IP, port; network) are not the
              > best solution. In our current implementation we can't do this because the SW load
              > balancer is based on filtering IP packets on the driver level.
              >
              > Currently I'm more interested in understanding whether it can our SW load balancer
              > can work with your replication at all?
              >
              > What makes me think that it could work is that in WLS6.0 a session failed over
              > to any cluster node can recover the replicated session.
              >
              > Can there be a problem with the cookies?
              > - are the P/S for replication put in the cookie by the node itself or by the proxy/HW
              > load balancer?
              >
              > >
              > >The options are -Dweblogic.debug.DebugReplication=true and
              > >-Dweblogic.debug.DebugReplicationDetails=true
              > >
              >
              > Great, thanks!
              >
              > Regards,
              > Frank Olsen
              

  • Distributed naming services for load balancing

    Can distributed naming services can be used for load balancing?

    ata,
    Before jumping to anything so bloated and limited as cougaar, take time to consider what you really need. Before grabing at the fanciest Java features like RMI, JINI, and custom SocketFactories; focus on what you are trying to accomplish.
    There are plenty of great answers right here, at this forum.
    Good hunting,
    John

  • Distributed HA cluster with load-balancing and failover: advice?

    My workplace has a Xeon Xserve, which acts as our primary external server, with an attached ActiveStorage XRAID. We have just purchased a second Xserve/XRAID set to act as a mirror, which we will colocate. Both have Leopard Server installed, along with an array of additional software.
    What we want to do is have both servers load-balance between the two, with failover in case of a server or XRAID fault. I plan on using RSYNC to mirror static files between the two, and I'm looking into PostgreSQL replication and load-balancing solutions for our database. I gather that Apache supports web-server failover and load-balancing, as well. But, that still leaves the actual host and network setup to arrange.
    Does Leopard server support such a thing? The only information I found on IP failover instructs the user to place the two servers on the same subnet, directly connected via ethernet cable; obviously, this would not work in my case.
    Ideally, what we'd end up with is a situation in which the two systems kept each other in sync, both in static files and database data, and load-balanced between themselves; in cases of failure, the remaining system would transparently assume all duties until the other was restored, at which time they would resynchronize
    Any suggestions on how I could arrange such a thing?

    Interesting. Does this DNS-based approach support session tracking, though? I would need to have a user directed to just one of the two servers for the duration of their session, to avoid having to synchronize temporary files and such.
    You can't have it both ways. You need to build tolerance into the app.
    At the simplest level where you run all traffic to one site and use the second site as a failover/standby site you'll be OK most of the time - all users will go to the same server and their sessions will be intact.
    However, under any failover situation (your primary site is down for some reason), there is going to be some level of session traffic that it going to switch over to the other site. If your site depends on sessions then you're going to need to tolerate this kind of situation - your app will need to be able to fail gracefully if a user comes in with an invalid session cookie.
    Note, though, that this may be less of an issue than you at first think - all DNS clients will cache DNS data for whatever TTL you set. This means that if a user looks up your site name and you return an IP address with a 30 minute TTL, then that user is going to use the same IP address for the next 30 minutes and isn't going to ask the server again. This should negate most chance of a user suddenly switching from one server location to the other in mid-session.
    The trick comes in setting the DNS TTL low enough to effect a failover, yet not so long that you impact performance - e.g. you don't want the user to perform a DNS lookup on every page load. You may find that 10 minutes is appropriate. Just bear in mind that this affects how long a user could see your site 'down' before the failover DNS kicks in. Clearly you don't want to set the DNS TTL to a day since that may prevent the user switching to the secondary site for 24 hours by which time, hopefully, the primary site is back up, anyway.
    The 'right' TTL value may take some analysis on your traffic to see how long a typical user 'session' is. If the average user spends 20 minutes on your site, then it would make sense to set your TTL to somewhere around 20 minutes to give the best chance of their entire session staying on the same server.

  • ALBPM - Batch events and Load Balancing

    Hi,
    We are planning to design a BPM solution for one of our current applications. The need we have is that the BPM solution should be able to start process instances in a batch mode. We receive about 25,000 to 30,000 events in a batch file and we need to start one BPM process instance for each record. Currently we are evaluating the ALBPM, but trying to figure out a best approach to do this. Also I would like to know what will be be ther better options to configure load balance instances of BPM for this scenario. I could not find any documents from BEA to address these. Anybody tried or came across this situation? Any documents/examples that can answer or suggest options for the batch modes? Thanks in advance!

    First of all, load balancing applies at various levels.
    1) load balancing at the horizontal level is only achieved by creating a WLS cluster and deploying the BPM engine in the cluster.
    2) load balancing at the vertical level can be achieved by creating many BPM engines and deploying processes on each Engine, this will distribute instances between the engines.
    if you need to create 25k instances of the same process in a batch, there is no problem, it will queue the executions and dispatch them to the available threads.
    A good approach would be to create instances in smaller chunks, to do this you can create a "batch process" that can get the file, split it in smaller chunks and process one chunk at a time. An important thing to consider when working with so many instances is that the transactions tend to get bigger and longer, so if you have a huge chunk, you can get a transaction timeout or a DB exception because of redo logs sizes.
    Hope this helps!
    MAriano Benitez
    Join us at BEAParticipate, May 6-9 2007 | Atlanta, Georgia

Maybe you are looking for