XA RESOURCE NOT RESPONDING IN PROD

Question

Hey guys ...I have seen posts for this kind of errors ..but didn't find any reason
for what is causing this problem...basically once this error starts happening the
server becomes unresponsive...any help is appreciated as this is happening in production
and the only way out seems to be restarting the server during business hrs which
is not a good idea...
####<May 20, 2002 2:58:35 PM EDT> <Warning> <JTA> <nynwspap1> <Cluster1> <Thread-0>
<> <> <110030> <XA resource [weblogic.jdbc.jts.Connection] has not responded in the
last 120 second(s).> ####<May 20, 2002 3:28:44 PM EDT> <Warning> <JTA> <nynwspap1>
<Cluster1> <Thread-0> <> <> <110030> <XA resource [weblogic.jdbc.jts.Connection]
has not responded in the
last 120 second(s).> ####<May 20, 2002 3:39:18 PM EDT> <Debug> <BofaApp> <nynwspap1>
<Cluster1> <ExecuteThread: '8' for queue: 'default'> <> <> <000000> <getLogicalData
for:CoreInfo> ####<May 20, 2002 3:39:18 PM EDT> <Error> <BofaApp> <nynwspap1> <Cluster1>
<ExecuteThread: '11' for queue: 'default'> <marcme> <> <000000> < AcctApproveHelper::approve()
Remote call was not successful java.rmi.RemoteException: EJB Exception:; nested exception
is: java.rmi.RemoteException: Message was not sent because transaction is not active.
transaction=(IdHash=3308339,Name = [EJB AccountBeanImpl.approveAccount()],Xid= 10531:c004c7fd25fa6b58,Status=Rolled
back. [Reason = weblogic.transaction.internal.TimedOutException: Transaction timed
out after 99seconds tx = transaction=(IdHash=3308339,Name = [EJB AccountBeanImpl.approveAccount()],Xid=10531:c004c7fd25fa6b58,Status=Active,numRepliesOwedMe=1,numRepliesOwedOthers=1,second
s since begin=99,seconds left=60,activeThread=Thread[ExecuteThread: '11' for queue:
'default',5,Thread Group for Queue: 'default'],ServerResourceInfo[weblogic.jdbc.jts.
Connection]=(state=suspended,assigned=none),SCInfo[Cluster1]=(state=active),properties=({weblogic.transaction.name=[EJB
AccountBeanImpl.approveAccount()], weblogic.jdbc =t3://10.215.7.169:7003}))],numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds
since begin=3213,seconds left=22,activeThread=Thread[ExecuteThread: '11' for queue:
'defau lt',5,Thread Group for Queue: 'default'],ServerResourceInfo[weblogic.jdbc.jts.Connection]=(state=rolledback,assigned=Cluster1),SCInfo[Cluster1]=(state=rolledback),prope
rties=({weblogic.transaction.name=[EJB AccountBeanImpl.approveAccount()], weblogic.jdbc=t3://10.215.7.169:7003}))
java.rmi.RemoteException: Message was not sent because transaction is not active.
transaction=(IdHash=3308339,Name = [EJB AccountBeanImpl.approveAccount()],Xid=10531:c0
04c7fd25fa6b58,Status=Rolled back. [Reason = weblogic.transaction.internal.TimedOutException:
Transaction timed out after 99seconds tx = transaction=(IdHash=3308339,Name = [EJB
AccountBeanImpl.approveAccount()],Xid=10531:c004c7fd25fa6b58,Status=Active,numRepliesOwedMe=1,numRepliesOwedOthers=1,second
s since begin=99,seconds left=60,activeThread=Thread[ExecuteThread: '11' for queue:
'default',5,Thread Group for Queue: 'default'],ServerResourceInfo[weblogic.jdbc.jts.
Connection]=(state=suspended,assigned=none),SCInfo[Cluster1]=(state=active),properties=({weblogic.transaction.name=[EJB
AccountBeanImpl.approveAccount()], weblogic.jdbc =t3://10.215.7.169:7003}))],numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds
since begin=3213,seconds left=22,activeThread=Thread[ExecuteThread: '11' for queue:
'defau lt',5,Thread Group for Queue: 'default'],ServerResourceInfo[weblogic.jdbc.jts.Connection]=(state=rolledback,assigned=Cluster1),SCInfo[Cluster1]=(state=rolledback),prope
rties=({weblogic.transaction.name=[EJB AccountBeanImpl.approveAccount()], weblogic.jdbc=t3://10.215.7.169:7003}))
at weblogic.transaction.internal.TransactionManagerImpl.sendRequest(TransactionManagerImpl.java:402)
at weblogic.rmi.internal.ServerRequest.sendOneWayRaw(ServerRequest.java:78) at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:108)
at com.symphoni.bofa.business.entity.qualification.QualificationBeanEOImpl_WLStub.setQualificationData(QualificationBeanEOImpl_WLStub.java:576)
at com.symphoni.bofa.business.session.accountmgr.table.QualificationTable.updateDatabase(QualificationTable.java:79)
at com.symphoni.bofa.business.session.accountmgr.table.TableOneToOne.save(TableOneToOne.java:54)
at com.symphoni.bofa.business.session.accountmgr.helper.BaseQualificationHelper.commit(BaseQualificationHelper.java:95)
at com.symphoni.bofa.business.session.accountmgr.SubmitHelper.saveToDatabase(SubmitHelper.java:96)
at com.symphoni.bofa.business.session.accountmgr.SubmitHelper.commit(SubmitHelper.java:75)
at com.symphoni.bofa.business.session.accountmgr.BaseAccount.submitAccount(BaseAccount.java:287)
at com.symphoni.bofa.business.session.accountmgr.AccountBeanImpl.submitAccount(AccountBeanImpl.java:150)
at com.symphoni.bofa.business.session.accountmgr.BaseAccount.approveAccount(BaseAccount.java:618)
at com.symphoni.bofa.business.session.accountmgr.AccountBeanImpl.approveAccount(AccountBeanImpl.java:950)
at com.symphoni.bofa.business.session.accountmgr.AccountBeanEOImpl.approveAccount(AccountBeanEOImpl.java:1564)
at com.symphoni.bofa.presentation.accountmgr.util.AcctApproveHelper.approve(AcctApproveHelper.java:167)
at com.symphoni.bofa.presentation.accountmgr.util.AcctApproveHelper.init(AcctApproveHelper.java:80)
at com.symphoni.bofa.presentation.accountmgr.servlet.BaseAcctController.processInit(BaseAcctController.java:564)
at com.symphoni.bofa.presentation.accountmgr.servlet.BaseAcctController.doPost(BaseAcctController.java:359)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:213)
at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:157)
at weblogic.servlet.jsp.PageContextImpl.forward(PageContextImpl.java:111) at jsp_servlet._view._pages._account._distributor._index._jspService(_index.java:327)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27) at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:213)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:1265)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:1622)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:137) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

Answer 1

Hi Krish
From your note, I am guessing you are using the Oracle thin driver with
WebLogic Server 6.0 (SP?). Have you tried updating the oracle thin driver to
the the latest version from oracle. They made quite a few fixes to the
driver. Older version 8.1.6 driver had problems and could take the server
into a hang. If you are not already using 8.1.7 or 9.0.1 driver I recommend
you to move to that version.
When the server goes unresponsive thread dumps will help us figure out what
makes the server unresponsive. This is a general thing to debug the issue.
Even if its intermittent, the next time it happens I recommend you make sure
you have at least 2 Full thread dumps.
Regards
sree
"Krishnan Venkataraman" <[email protected]> wrote in message
news:[email protected]...
>
>
Sree ..I was warned that this wud be the typical reply from bea supportusually for
this problem...
firstly...this problem happens intermittently...in that case I do not seehow I can
take these thread dumps..since this beviour is unexpected...
I use classes12.zip and the patch CR047274_60.jar which weblogic providesfor the
"broken pipe" bug..
When this event occured..I checked the pool size and it was far less thanthe max
size ...and this error seems to be rendering the server unusable but for arestart...!
>
Thanx,
Krish
"Sree Bodapati" <[email protected]> wrote:
Also what driver are you using to connect to the database? (to create
connection pools)
sree
"KRISHNAN VENKATARAMAN" <[email protected]> wrote in message
news:[email protected]...
Hey guys ...I have seen posts for this kind of errors ..but didn't findany reason
for what is causing this problem...basically once this error startshappening the
server becomes unresponsive...any help is appreciated as this is
happening
in production
and the only way out seems to be restarting the server during businesshrs
which
is not a good idea...
####<May 20, 2002 2:58:35 PM EDT> <Warning> <JTA> <nynwspap1>
<Cluster1>
<Thread-0>
<> <> <110030> <XA resource [weblogic.jdbc.jts.Connection] has notresponded in the
last 120 second(s).> ####<May 20, 2002 3:28:44 PM EDT> <Warning> <JTA><nynwspap1>
<Cluster1> <Thread-0> <> <> <110030> <XA resource[weblogic.jdbc.jts.Connection]
has not responded in the
last 120 second(s).> ####<May 20, 2002 3:39:18 PM EDT> <Debug>
<BofaApp>
<nynwspap1>
<Cluster1> <ExecuteThread: '8' for queue: 'default'> <> <> <000000><getLogicalData
for:CoreInfo> ####<May 20, 2002 3:39:18 PM EDT> <Error> <BofaApp><nynwspap1> <Cluster1>
<ExecuteThread: '11' for queue: 'default'> <marcme> <> <000000> <AcctApproveHelper::approve()
Remote call was not successful java.rmi.RemoteException: EJB
Exception:;
nested exception
is: java.rmi.RemoteException: Message was not sent because transactionis
not active.
transaction=(IdHash=3308339,Name = [EJBAccountBeanImpl.approveAccount()],Xid=10531:c004c7fd25fa6b58,Status=Rolled
back. [Reason = weblogic.transaction.internal.TimedOutException:
Transaction timed> >> out after 99seconds tx = transaction=(IdHash=3308339,Name = [EJB>>AccountBeanImpl.approveAccount(),Xid=10531:c004c7fd25fa6b58,Status=Active,n
umRepliesOwedMe=1,numRepliesOwedOthers=1,second
s since begin=99,seconds left=60,activeThread=Thread[ExecuteThread:
'11'
for queue:
'default',5,Thread Group for Queue:'default'],ServerResourceInfo[weblogic.jdbc.jts.
>Connection]=(state=suspended,assigned=none),SCInfo[Cluster1]=(state=active),
properties=({weblogic.transaction.name=[EJB
AccountBeanImpl.approveAccount()], weblogic.jdbc
=t3://10.215.7.169:7003}))],numRepliesOwedMe=0,numRepliesOwedOthers=0,second
s
since begin=3213,seconds left=22,activeThread=Thread[ExecuteThread:
'11'
for queue:
'defau lt',5,Thread Group for Queue:
'default'],ServerResourceInfo[weblogic.jdbc.jts.Connection]=(state=rolledbac
k,assigned=Cluster1),SCInfo[Cluster1]=(state=rolledback),prope
rties=({weblogic.transaction.name=[EJB
AccountBeanImpl.approveAccount()],
weblogic.jdbc=t3://10.215.7.169:7003}))
java.rmi.RemoteException: Message was not sent because transaction isnot
active.
transaction=(IdHash=3308339,Name = [EJBAccountBeanImpl.approveAccount()],Xid=10531:c0
04c7fd25fa6b58,Status=Rolled back. [Reason =
weblogic.transaction.internal.TimedOutException:> >> Transaction timed out after 99seconds tx => >transaction=(IdHash=3308339,Name = [EJB> >>>>AccountBeanImpl.approveAccount(),Xid=10531:c004c7fd25fa6b58,Status=Active,
n
umRepliesOwedMe=1,numRepliesOwedOthers=1,second
s since begin=99,seconds left=60,activeThread=Thread[ExecuteThread:
'11'
for queue:
'default',5,Thread Group for Queue:'default'],ServerResourceInfo[weblogic.jdbc.jts.
>Connection]=(state=suspended,assigned=none),SCInfo[Cluster1]=(state=active),
properties=({weblogic.transaction.name=[EJB
AccountBeanImpl.approveAccount()], weblogic.jdbc
=t3://10.215.7.169:7003}))],numRepliesOwedMe=0,numRepliesOwedOthers=0,second
s
since begin=3213,seconds left=22,activeThread=Thread[ExecuteThread:
'11'
for queue:
'defau lt',5,Thread Group for Queue:
'default'],ServerResourceInfo[weblogic.jdbc.jts.Connection]=(state=rolledbac
k,assigned=Cluster1),SCInfo[Cluster1]=(state=rolledback),prope
rties=({weblogic.transaction.name=[EJB
AccountBeanImpl.approveAccount()],
weblogic.jdbc=t3://10.215.7.169:7003}))
at
weblogic.transaction.internal.TransactionManagerImpl.sendRequest(Transaction
ManagerImpl.java:402)
atweblogic.rmi.internal.ServerRequest.sendOneWayRaw(ServerRequest.java:78)
at
weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:108)
at
com.symphoni.bofa.business.entity.qualification.QualificationBeanEOImpl_WLSt
ub.setQualificationData(QualificationBeanEOImpl_WLStub.java:576)
at
com.symphoni.bofa.business.session.accountmgr.table.QualificationTable.updat
eDatabase(QualificationTable.java:79)
at
com.symphoni.bofa.business.session.accountmgr.table.TableOneToOne.save(Table
OneToOne.java:54)
at
com.symphoni.bofa.business.session.accountmgr.helper.BaseQualificationHelper
..commit(BaseQualificationHelper.java:95)
at
com.symphoni.bofa.business.session.accountmgr.SubmitHelper.saveToDatabase(Su
bmitHelper.java:96)
at
com.symphoni.bofa.business.session.accountmgr.SubmitHelper.commit(SubmitHelp
er.java:75)
at
com.symphoni.bofa.business.session.accountmgr.BaseAccount.submitAccount(Base
Account.java:287)
at
com.symphoni.bofa.business.session.accountmgr.AccountBeanImpl.submitAccount(
AccountBeanImpl.java:150)
at
com.symphoni.bofa.business.session.accountmgr.BaseAccount.approveAccount(Bas
eAccount.java:618)
at
com.symphoni.bofa.business.session.accountmgr.AccountBeanImpl.approveAccount
(AccountBeanImpl.java:950)
at
com.symphoni.bofa.business.session.accountmgr.AccountBeanEOImpl.approveAccou
nt(AccountBeanEOImpl.java:1564)
at
com.symphoni.bofa.presentation.accountmgr.util.AcctApproveHelper.approve(Acc
tApproveHelper.java:167)
at
com.symphoni.bofa.presentation.accountmgr.util.AcctApproveHelper.init(AcctAp
proveHelper.java:80)
at
com.symphoni.bofa.presentation.accountmgr.servlet.BaseAcctController.process
Init(BaseAcctController.java:564)
at
com.symphoni.bofa.presentation.accountmgr.servlet.BaseAcctController.doPost(
BaseAcctController.java:359)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760) atjavax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:213)
at
weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImp
l.java:157)
at
weblogic.servlet.jsp.PageContextImpl.forward(PageContextImpl.java:111)
atjsp_servlet._view._pages._account._distributor._index._jspService(_index.jav
a:327)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27) at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
:213)
at
weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletCo
ntext.java:1265)
at
weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java
:1622)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:137) atweblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)