Clustering with wl 6.1 - Questions Part II

Question

Again, here's the setup:
          Admin: 192.168.1.135:7001 (XP). Cluster address:192.168.1.239, 192.168.1.71
          managed server 1: 192.168.1.239:7001 (SunOS)
          managed server 2: 192.168.1.71:7001 ('98)
          I have a single SLSB deployed on the cluster. The deployment descriptor
          has been configured to stateless clustering. The bean has a single method
          (returnMessage(String msg)) that accepts the strings and returns it.
          A standalone java client acesses the cluster by specifying the JNDI URL
          "t3://192.168.1.71, 192.168.1.239:7001" and works properly. The java client
          does a JNDI lookup once, caches the home reference and call home.create in
          an infinite loop.
          1. After reading one of the posts here, I physically restarted one of the
          machines in the cluster (192.168.1.71). The java client was stuck for a 4 *
          30 secs. The managed server whose network cable I yanked out kept showing
          the following message every 30 secs (as the docs say):
          <25/08/2002 13:52:21> <Error> <Cluster> <Multicast socket receive error:
          java.io.InterruptedIOException: Receive timed out
          java.io.InterruptedIOException: Receive timed out
          at java.net.PlainDatagramSocketImpl.receive(Native Method)
          at java.net.DatagramSocket.receive(DatagramSocket.java:392)
          at weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:145)
          at
          weblogic.cluster.MulticastManager.execute(MulticastManager.java:293)
          at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:137)
          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
          justthe way it should be (hopefully)! Now, the real fun started when i put
          the cord back. The admin server should have detected that the managed
          server, right (The console should show the managed server as "Running" in
          the "Servers" tab)? Wrong, it doesn't! Why?
          2. I restarted the weblogic server on 192.168.1.71 and sure enough, it
          joined the cluster. After it joined the cluster, the last line of
          "weblogic.log" is:
          ####<Aug 25, 2002 2:15:53 PM PDT> <Info> <Management> <DUMMY135> <wlsdomain>
          <ExecuteThread: '8' for queue: 'default'> <system> <> <140009>
          <Configuration changes for domain saved to the repository.>
          And the "wl-domain.log" ("domain" is the name of my domain) has the last
          entry as:
          ####<Aug 25, 2002 1:43:56 AM PDT> <Notice> <WebLogicServer> <dummy71>
          <dummy71> <ExecuteThread: '14' for queue: 'default'> <system> <> <000332>
          <Started WebLogic Managed Server "dummy71" for domain "domain" running in
          Development Mode>
          The last few lines on the console window of the managed server (dummy71)
          shows:
          Starting Cluster Service ....
          <25/08/2002 14:13:55> <Notice> <WebLogicServer> <ListenThread listening on
          port 7001, ip address 192.168.1.71>
          <25/08/2002 14:13:56> <Notice> <Cluster> <Listening for multicast messages
          (cluster wlcluster) on port 7001 at address 237.0.0.1>
          <25/08/2002 14:13:56> <Notice> <WebLogicServer> <Started WebLogic Managed
          Server "dummy71" for domain "domain" running in Development Mode>
          2.1 I presume the "Configuration changes for domain saved to repository"
          indicates weblogic changed "cofig.xml"?
          2.2 Why would it update config.xml?
          2.3 Where is the entry in the logs to indicate that dummy71 joined the
          cluster?
          2.4 dummy71 joined the cluster at 14:13:56 while the admin server has
          the entry at 1:43:56 AM PDT. Why is there a timezone mentioned in
          "wl-domain.log"? Further investigation found out that time zone on the admin
          server was Pacific (US). Changed it to GMT+5:30 (so its the same on both),
          restarted weblogic on dummy71. Again, the logs mention PDT on the admin
          server, while there is no mention of the time zone on the managed server?
          Strange! Finally, the system clock on admin server shows 3:01 PM, while
          weblogic is about an hour behind in a different time zone (the last entry in
          the logs now has the timestamp Aug 25, 2002 2:02:14 AM PDT).
          3. Finally, i had modified my config.xml to add the following:
          <ServerDebug DebugCluster="true"
          DebugClusterAnnouncements="true"
          DebugClusterFragments="true" DebugClusterHeartbeats="true"
          Name="dummy239"/>
          I added the lines above for both the servers in the cluster to take a look
          at the chatter between the admin server and the managed servers
          (specifically the "heartbeat" every 30 seconds, the synchronization of
          clustered services (emphasis on EJB and JNDI)). Are the entries above
          correct and in the right place? If yes, what log files do i check for this?
          Can anyone tell me a sample message to look for (assuming this chatter is
          recorded in weblogic.log or wl-domain.log (the two log files i have)) so I
          can verify the same in my logs (and probably learn something from them
          With Warm Regards,
          Manav.
          

Answer 1

BTW - when I said "bad test", I didn't mean you should not do it, but rather
          that it is a "worst case scenario" (and something that BEA should be testing
          Peace,
          Cameron Purdy
          Tangosol, Inc.
          http://www.tangosol.com/coherence.jsp
          Tangosol Coherence: Clustered Replicated Cache for Weblogic
          "Manavendra Gupta" <[email protected]> wrote in message
          news:[email protected]...
          > Thanks for the reply Cameron. I conducted the same test by physically
          > shutting down one of the machines (which I believe simulates a crash) and
          > got the same results.
          >
          > And since I'm rather new to weblogic i've been really keen on getting the
          > answers to my queries :-)
          >
          > --
          > With Warm Regards,
          > Manav.
          >
          > "Cameron Purdy" <[email protected]> wrote in message
          > news:[email protected]...
          > > Plugging and unplugging a network cable is a very bad test. Some of the
          > JVM
          > > implementations do not re-establish multicast traffic if you do that, or
          > > they re-establish it only in one direction (it could be a bug in the OS
          or
          > > socket libs too). We implemented a solution for this exact problem, but
          it
          > > took a lot of research and coding.
          > >
          > > A better test would be to have a computer plugged to a switch plugged to
          a
          > > switch plugged to the other computer, then unplug the connection between
          > the
          > > switches.
          > >
          > > Peace,
          > >
          > > Cameron Purdy
          > > Tangosol, Inc.
          > > http://www.tangosol.com/coherence.jsp
          > > Tangosol Coherence: Clustered Replicated Cache for Weblogic
          > >
          > >
          > > "Manavendra Gupta" <[email protected]> wrote in message
          > > news:[email protected]...
          > > > Again, here's the setup:
          > > > Admin: 192.168.1.135:7001 (XP). Cluster address:192.168.1.239,
          > > 192.168.1.71
          > > > managed server 1: 192.168.1.239:7001 (SunOS)
          > > > managed server 2: 192.168.1.71:7001 ('98)
          > > >
          > > > I have a single SLSB deployed on the cluster. The deployment
          > descriptor
          > > > has been configured to stateless clustering. The bean has a single
          > method
          > > > (returnMessage(String msg)) that accepts the strings and returns it.
          > > >
          > > > A standalone java client acesses the cluster by specifying the JNDI
          URL
          > > > "t3://192.168.1.71, 192.168.1.239:7001" and works properly. The java
          > > client
          > > > does a JNDI lookup once, caches the home reference and call
          home.create
          > in
          > > > an infinite loop.
          > > >
          > > > 1. After reading one of the posts here, I physically restarted one of
          > the
          > > > machines in the cluster (192.168.1.71). The java client was stuck for
          a
          > 4
          > > *
          > > > 30 secs. The managed server whose network cable I yanked out kept
          > showing
          > > > the following message every 30 secs (as the docs say):
          > > > <25/08/2002 13:52:21> <Error> <Cluster> <Multicast socket receive
          error:
          > > > java.io.InterruptedIOException: Receive timed out
          > > > java.io.InterruptedIOException: Receive timed out
          > > > at java.net.PlainDatagramSocketImpl.receive(Native Method)
          > > > at java.net.DatagramSocket.receive(DatagramSocket.java:392)
          > > > at
          > > weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:145)
          > > > at
          > > > weblogic.cluster.MulticastManager.execute(MulticastManager.java:293)
          > > > at
          > weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:137)
          > > > at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
          > > >
          > > > justthe way it should be (hopefully)! Now, the real fun started when i
          > put
          > > > the cord back. The admin server should have detected that the managed
          > > > server, right (The console should show the managed server as "Running"
          > in
          > > > the "Servers" tab)? Wrong, it doesn't! Why?
          > > > 2. I restarted the weblogic server on 192.168.1.71 and sure enough, it
          > > > joined the cluster. After it joined the cluster, the last line of
          > > > "weblogic.log" is:
          > > > ####<Aug 25, 2002 2:15:53 PM PDT> <Info> <Management> <DUMMY135>
          > > <wlsdomain>
          > > > <ExecuteThread: '8' for queue: 'default'> <system> <> <140009>
          > > > <Configuration changes for domain saved to the repository.>
          > > >
          > > > And the "wl-domain.log" ("domain" is the name of my domain) has the
          last
          > > > entry as:
          > > > ####<Aug 25, 2002 1:43:56 AM PDT> <Notice> <WebLogicServer> <dummy71>
          > > > <dummy71> <ExecuteThread: '14' for queue: 'default'> <system> <>
          > <000332>
          > > > <Started WebLogic Managed Server "dummy71" for domain "domain" running
          > in
          > > > Development Mode>
          > > >
          > > > The last few lines on the console window of the managed server
          (dummy71)
          > > > shows:
          > > > Starting Cluster Service ....
          > > > <25/08/2002 14:13:55> <Notice> <WebLogicServer> <ListenThread
          listening
          > on
          > > > port 7001, ip address 192.168.1.71>
          > > > <25/08/2002 14:13:56> <Notice> <Cluster> <Listening for multicast
          > messages
          > > > (cluster wlcluster) on port 7001 at address 237.0.0.1>
          > > > <25/08/2002 14:13:56> <Notice> <WebLogicServer> <Started WebLogic
          > Managed
          > > > Server "dummy71" for domain "domain" running in Development Mode>
          > > >
          > > > 2.1 I presume the "Configuration changes for domain saved to
          > > repository"
          > > > indicates weblogic changed "cofig.xml"?
          > > > 2.2 Why would it update config.xml?
          > > > 2.3 Where is the entry in the logs to indicate that dummy71 joined
          > the
          > > > cluster?
          > > > 2.4 dummy71 joined the cluster at 14:13:56 while the admin server
          > has
          > > > the entry at 1:43:56 AM PDT. Why is there a timezone mentioned in
          > > > "wl-domain.log"? Further investigation found out that time zone on the
          > > admin
          > > > server was Pacific (US). Changed it to GMT+5:30 (so its the same on
          > both),
          > > > restarted weblogic on dummy71. Again, the logs mention PDT on the
          admin
          > > > server, while there is no mention of the time zone on the managed
          > server?
          > > > Strange! Finally, the system clock on admin server shows 3:01 PM,
          while
          > > > weblogic is about an hour behind in a different time zone (the last
          > entry
          > > in
          > > > the logs now has the timestamp Aug 25, 2002 2:02:14 AM PDT).
          > > >
          > > > 3. Finally, i had modified my config.xml to add the following:
          > > > <ServerDebug DebugCluster="true"
          > > > DebugClusterAnnouncements="true"
          > > > DebugClusterFragments="true" DebugClusterHeartbeats="true"
          > > > Name="dummy239"/>
          > > > I added the lines above for both the servers in the cluster to take a
          > look
          > > > at the chatter between the admin server and the managed servers
          > > > (specifically the "heartbeat" every 30 seconds, the synchronization of
          > > > clustered services (emphasis on EJB and JNDI)). Are the entries above
          > > > correct and in the right place? If yes, what log files do i check for
          > > this?
          > > > Can anyone tell me a sample message to look for (assuming this chatter
          > is
          > > > recorded in weblogic.log or wl-domain.log (the two log files i have))
          so
          > I
          > > > can verify the same in my logs (and probably learn something from them
          > > > :-) ).
          > > >
          > > > --
          > > > With Warm Regards,
          > > > Manav.
          > > >
          > > >
          > > >
          > >
          > >
          >
          >