Weblogic portal is getting hanged after a regular interval

Question

Hi,
We are using WLP 8.1 SP 5 on a HP Unix environment. Our Weblogic instance is getting hanged and we need to restart the same. This problem is happing once in a 3 weeks time. From console we cheked that avilable memory for JVM heap is quite low when this happens. Below are the some of the log messages we have found in the weblogic log:
<Jun 28, 2010 6:12:59 AM GMT> <Error> <HTTP> <BEA-101083> <Connection failure.
java.net.SocketException: Error in poll for fd=172, revents=32
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:131)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
>
<Jun 28, 2010 6:45:14 AM GMT> <Error> <WebLogicServer> <BEA-000337> <ExecuteThread: '57' for queue: 'default' has been busy for "966" seconds working on the request "Http Reques
t: /Xxxxx/global/docs/en/careers/Xxxxx_Power_Source.wmv", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
********* resources error jsp is called *******
<Jun 28, 2010 7:55:14 AM GMT> <Error> <WebLogicServer> <BEA-000337> <ExecuteThread: '13' for queue: 'default' has been busy for "606" seconds working on the request "Http Reques
t: /Xxxxx/global/docs/en/xxxxx/media_publications/technical_papers/xxxx.pdf", which is more than the configured time (StuckThreadMaxTime) of "600" seco
nds.>
********* resources error jsp is called *******
<Jun 28, 2010 7:46:13 AM GMT> <Error> <HTTP> <BEA-101046> <[ServletContext(id=12830383,name=xxxxxportal,context-path=)] mt:mail: Could not send message
javax.mail.SendFailedException: No recipient addresses
at javax.mail.Transport.send0(Transport.java:111)
at javax.mail.Transport.send(Transport.java:81)
at org.apache.taglibs.mailer2.BackgroundMailThread.run(BackgroundMailThread.java:65)
     ********* resources error jsp is called *******
Any help is highly appriciated.

Answer 1

Hi,
Please find below my findings about the memory leak.
Many (around 40 where max connection is 50) connection were created without they getting closed in a very short span of time. This caused the connection pool to be filled as rest of the connections in the connection pool were doing something else. Below is the log message:
####<Jul 4, 2010 12:24:01 AM GMT> <Info> <Common> <domainname.xxxxxxxx.com> <EXL> <ExecuteThread: '4' for queue: 'default'> <<anonymous>> <BEA1-39390F6683087D485CB1> <BEA-000627> <Reached maximum capacity of pool "portalPool", making "0" new resource instances instead of "1".>
####<Jul 4, 2010 12:24:01 AM GMT> <Info> <Common> <domainname.xxxxxxxx.com> <EXL> <ExecuteThread: '48' for queue: 'default'> <<anonymous>> <BEA1-393D0F6683087D485CB1> <BEA-000627> <Reached maximum capacity of pool "portalPool", making "0" new resource instances instead of "1".>
####<Jul 4, 2010 12:26:06 AM GMT> <Info> <Common> <domainname.xxxxxxxx.com> <EXL> <ExecuteThread: '45' for queue: 'default'> <<anonymous>> <BEA1-397A0F6683087D485CB1> <BEA-000627> <Reached maximum capacity of pool "portalPool", making "0" new resource instances instead of "1".>
This caused all the threads handling request to the application hanged.
####<Jul 4, 2010 12:29:24 AM GMT> <Error> <WebLogicServer> <domainname.xxxxxxxx.com> <EXL> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '1' for queue: 'default' has been busy for "636" seconds working on the request "Http Request: /xxxxxxxx.portal", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<Jul 4, 2010 12:29:24 AM GMT> <Error> <WebLogicServer> <domainname.xxxxxxxx.com> <EXL> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '2' for queue: 'default' has been busy for "696" seconds working on the request "Http Request: /xxxxxxxx.portal", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
69 of 70 threads hanged and logged this error message
Also, other Weblogic modules (EJBs etc.) were not able to release connections due to this connection leak problem.
<Jul 4, 2010 12:31:13 AM GMT> <Info> <EJB> <domainname.xxxxxxxx.com> <EXL> <ExecuteThread: '15' for queue: 'default'> <<anonymous>> <BEA1-38C90F6683087D485CB1> <BEA-010051> <EJB Exception occurred during invocation from home: [email protected]106df27 threw exception: javax.ejb.TransactionRolledbackLocalException: EJB Exception: ; nested exception is: javax.ejb.TransactionRolledbackLocalException: EJB Exception: ; nested exception is: weblogic.transaction.internal.TimedOutException: Transaction timed out after 502 seconds
BEA1-38C90F6683087D485CB1
These hanged threads and EJBs caused the memory leak in JVM heap.
####<Jul 4, 2010 2:00:35 AM GMT> <Error> <T3Misc> <domainname.xxxxxxxx.com> <EXL> <weblogic.time.TimeEventGenerator> <<WLS Kernel>> <> <BEA-000712> <Exception invoking Scheduled Trigger
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
When checked the JDBC log the below exception was found:
SQLException: SQLState(null) vendor code(17002)
SQLException: SQLState(null) vendor code(17002)
java.sql.SQLException: Io exception: Broken pipe (errno:32)
     at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
     at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
     at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
     at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
So, what might cause this JDBC error. Is the problem at Weblogic end or the database end? It seems that the connections are hanging due to this JDBC error which is causing the connection pool to be filled up. Subsequently threads are hanging which leads to memory leak.
Please help as this issue is reoccuring again and again!!!