Failure on one Server casues other to lock

Question

          Greetings,
          I am running a WLS 6.1 sp2 cluster with 2 managed server instances on 2 W2K servers.
          When one server in the cluster fails (eg java.lang.OutOfMemoryError) and stops responding,
          the other server in the cluster will not respond to requests (as viewed on the console,
          request queue applet) until the failed server is restarted. Does anyone know why?
          Thanks
          Mike Horn
          

Answer 1

Thanks for the quick response.
          I took many thread dumps, I took them out of the file because it was upward
          of 5M
          The dumps after the server stopped taking new requests didn't move, all of
          the thread dumps look the same as what I put into the zips. I can upload
          the full files if you like, but there are dumps from after startup, during
          the test run and after it was hung. It is hard to tell what is what other
          than the last few were from after the hung state.
          The reason there are some threads are waiting in our code (actually
          inetsoft's code) is because there is a syncronized method in their
          PDF generation code. It is 3rd party so we are not able to "fix" it.
          The server is really in a hung state. We can monitor all database activity,
          and at the point it no longer takes any new requests we see no database
          activity and we are unable to do a weblogic.Admin ping, or connect to the
          server instance in any way (directly to 7001 bypassing the proxy does NOT
          work either).
          We do not get the out of memory errors when we run with the hotspot server
          VM, however since there is a bug in the VM we are not able to take thread
          dumps running with it. We were forced to run with the classic VM during our
          tests to get good thread dumps, which cause our OOM errors since Xincgc
          and other memory options do not work with the classic VM.
          Thanks,
          Joel
          "Kumar Allamraju" <[email protected]> wrote in message
          news:[email protected]...
          > Look at the thread dump.
          > 14 out of 26 execute threads are struck in your code. Can you
          > investigate further and see why it's happening like that.
          >
          > Also you may want to take atleast 3 thread dumps and see if there's any
          > progress in thread's state.
          >
          > Some threads are in waitForData(ResponseImpl.java) call , which are
          > waiting for the response from the other server and the other server
          > turned out be running OOM. So at this point i'm not really seeing the
          > server hang.
          >
          > Few things we should try:
          >
          > 1. Investigate why you are seeing OOM? Fix it
          > 2. Create a seperate ExecuteThread Queue for your webapp and route the
          > webapp traffic via this queue. See our docs.
          >
          > Let us know how it goes.
          >
          > --
          > Kumar
          >
          > Joel Caven wrote:
          >
          > > We are having the same problem, see the attached zip for the thread dump
          and
          > > stderr messages.
          > > 2 node cluster on WL61 SP2 on W2K. Admin ping fails once the out of
          memory
          > > happens. Admin
          > > console is unable to attach to the server until it is rebooted. We have
          > > tried with and without hotspot,
          > > it happens in both cases. We have also tried with and without NativeIO
          on,
          > > and also with the SP2
          > > NativeIO patch for W2K. In all cases the server fails to respond after
          the
          > > out of memory error.
          > >
          > > Thanks,
          > > Joel.
          > >
          > > P.S. We normally have to restart all members of our cluster to fix the
          > > problem, sometimes I even have to restart the admin server.
          > >
          > >
          > > "Kumar Allamraju" <[email protected]> wrote in message
          > > news:[email protected]...
          > >
          > >>How about taking a thread dump when the server is unresponsive?
          > >>
          > >>Mike Horn wrote:
          > >>
          > >>
          > >>>Yep, its true. No reply to PING until the other server is restarted.
          > >>>
          > > Sometime, like
          > >
          > >>>I mentioned, both need to be restarted.
          > >>>
          > >>>mike
          > >>>
          > >>>Kumar Allamraju <[email protected]> wrote:
          > >>>
          > >>>>That's not true.
          > >>>>May be the requests have been piled up and you see the other server
          > >>>>may be slow or unresponsiveness.
          > >>>>
          > >>>>Can you try "java weblogic.Admin url user password PING" on the live
          > >>>>server and see if you get a response or not?
          > >>>>
          > >>>>--
          > >>>>Kumar
          > >>>>
          > >>>>Mike Horn wrote:
          > >>>>
          > >>>>
          > >>>>>Greetings,
          > >>>>>
          > >>>>>I am running a WLS 6.1 sp2 cluster with 2 managed server instances on
          > >>>>>
          > >>>>2 W2K servers.
          > >>>>
          > >>>>> When one server in the cluster fails (eg java.lang.OutOfMemoryError)
          > >>>>>
          > >>>>and stops responding,
          > >>>>
          > >>>>>the other server in the cluster will not respond to requests (as
          > >>>>>
          > > viewed
          > >
          > >>>>on the console,
          > >>>>
          > >>>>>request queue applet) until the failed server is restarted. Does
          > >>>>>
          > > anyone
          > >
          > >>>>know why?
          > >>>>
          > >>>>>Thanks
          > >>>>>Mike Horn
          > >>>>>
          > >>>>>
          > >
          > >
          >