Out Of Memory Error - Weblogic server

Question

Hi,
We are facing following errors in our production environment frequently...
Please guide us how can we identify root cause.. because every time this error come in different functionalities so we can not blame one functionality (Code)...
Error # 1
======
<Jun 18, 2009 4:24:41 PM IST> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: .
java.lang.OutOfMemoryError:
>
<Jun 18, 2009 4:24:45 PM IST> <Error> <Server> <BEA-002608> <The ListenThread because of an error: java.lang.OutOfMemoryError: memsystemAllocObject - Object size: 24
java.lang.OutOfMemoryError: memsystemAllocObject - Object size: 24
     at java.net.PlainSocketImpl.socketAccept(Native Method)
     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
     at java.net.ServerSocket.accept(ServerSocket.java:421)
     at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:34)
     Truncated. see log file for complete stacktrace
>
Error # 2
======
<Jun 18, 2009 7:32:27 PM IST> <Error> <Server> <BEA-002608> <The ListenThread because of an error: java.lang.OutOfMemoryError:
java.lang.OutOfMemoryError:
>
<Jun 18, 2009 7:32:27 PM IST> <Error> <Server> <BEA-002608> <The ListenThread because of an error: java.lang.OutOfMemoryError: memsystemAllocObject - Object size: 24
java.lang.OutOfMemoryError: memsystemAllocObject - Object size: 24
     at java.net.PlainSocketImpl.socketAccept(Native Method)
     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
     at java.net.ServerSocket.accept(ServerSocket.java:421)
     at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:34)
     Truncated. see log file for complete stacktrace
>
Thanks in Advance.
Edited by: Rushang.Kansara on Jul 1, 2009 6:46 AM

Answer 1

Once you get to the state where you're out of memory, you'll get any number of possible variations on the error message. These variations will give you absolutely zero help for why you've run out of memory. Without more information, it's impossible to tell whether you have a memory leak, or whether you simply have too small a value for "maxmem" in the JVM.
Typically, I would assume that if the server has been running for a "while" (whatever that means) without error, then it probably isn't a simple issue with setting the "maxmem" value high enough. If it does fail pretty quickly after startup (and processing of initial messages), then it very well could be a maxmem problem. This is done by adding or changing the "-Xmxnnn" command-line parameter to the JVM. For instance, if you wanted to give it 2g of max heap, you would put "-Xmx2048m" on the command line. That value won't work on Windows (not sure about Vista or 64-bit), but it will work perfectly well on Unix/Linux.
If you have a memory leak, it's good to first verify this by sitting on the "Server"->"Performance" page in the WebLogic console. If you watch the "memory free" (or "memory inuse") percentage, you'll see that it varies by a wide margin over time. What you want to do is to be able to force a GC (there should still be a button there to do that) and then note the percent free immediately after that. You should do this every few minutes while the server is actively working. If the percent free value (after the GC) is consistently going down, even slowly, then you likely have a memory leak. This is where it gets a little more difficult. You could look at the free VisualVM tool which provides some memory debugging features, but they are somewhat limited (I only know this from reading some literature, I haven't tried it), or you could look at the commercial tools JProfiler, JProbe, or YourKit (in decreasing order of my personal preference).