XAER_RMFAIL : Resource manager is unavailable

Question

Weblogic 8.1, I don't recall which service pack, sp10? Maybe?
Anyway, here's the relevant stack trace. We had a strange WLS implosion last
night that we're tracking down. We saw these when our code was trying to get
a connection out of the datapool.
<Mar 1, 2006 10:09:12 PM PST> <Warning> <JTA> <BEA-110030> <XA resource
[weblogic.jdbc.wrapper.JTSXAResourceImpl] has not responded in the last 120
second(s).>
01 Mar 2006 22:09:21,652 WARN com.pfizer.ecms.as.MetricsSession - logEvent:
exception: start() failed on resource
'weblogic.jdbc.wrapper.JTSXAResourceImpl': XAER_RMFAIL : Resource manager is
unavailable
javax.transaction.xa.XAException: Internal error: XAResource
'weblogic.jdbc.wrapper.JTSXAResourceImpl' is unavailable
at
weblogic.transaction.internal.XAResourceDescriptor.checkResource(XAResourceD
escriptor.java:1019)
at
weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResour
ceDescriptor.java:572)
at
weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInf
o.java:1068)
at
weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceI
nfo.java:1001)
at
weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceIn
fo.java:203)
at
weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTra
nsactionImpl.java:419)
at weblogic.jdbc.jts.Driver.createLocalConnection(Driver.java:207)
at weblogic.jdbc.jts.Driver.connect(Driver.java:154)
at
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java
:298)
at
com.pfizer.ecms.asutil.DataAccessManager.getConnection(DataAccessManager.jav
a:117)
Here's our configuration:
<JDBCConnectionPool CapacityIncrement="1"
DriverName="oracle.jdbc.driver.OracleDriver" InitialCapacity="1"
MaxCapacity="150" Name="OraclePool"
Password="*****"
Properties="user=******" ShrinkPeriodMinutes="5"
Targets="myserver" TestConnectionsOnCreate="true"
TestConnectionsOnRelease="true" TestConnectionsOnReserve="true"
TestTableName="dual" URL="jdbc:oracle:thin:@garrison.back:1521:IC"/>
<JDBCTxDataSource EnableTwoPhaseCommit="true"
JNDIName="dbpool" Name="OracleSource"
PoolName="OraclePool" Targets="myserver"/>
Looking this up, I was pointed to a JTA FAQ page on edocs, and it basically
said we should be Refreshing the connection pool, but the RefreshMinutes
parameter is deprecated, and as you can see we're testing connections as is.
We're puzzled by a couple of things, almost certainly application related,
but still relevant. We have 150 connections allocated to the pool, but even
at peak load during the day we bounce around the 40 mark. When we got the
above exception, we were hitting the 150 ceiling. We don't quite know why
yet.
During this time, our DB server was pretty heaviliy loaded. There were some
horrendous queries running, some data imports, etc. Bascially a bunch of
stuff that happens at the beginning of the month. However, there weren't
necessarily a lot of actual users on the system, just a select few.
One of the things that was happening was a user was using the web interface,
and sent off a huge horrible query. When it didn't come back in a reasonable
amount of time, he simply hit "Refresh" on the page, thereby submitting the
query again (thank you Sir, may I have another!). This was partly why the
database was so heavily loaded.
Now, we're puzzled by a couple of things. One, is the spike of actual
connections in the DB Pool up to 150. This makes no sense.
Second, is this error detailed above. I've tried some contrived
experiements, and I can not raise this specific exception myself. My
experiment was to issue a "update table set column = 1 where id = 1" in a
generic SQL browser, thus locking the row, and then issue the identical
command through a Session Bean call. It has happily sat there for up to 10
minutes (600 seconds, which is our default <JTA Name="inf4.2"
TimeoutSeconds="600"/> setting), and it eventually times out, but not with
the same exception. Rather I get "java.sql.SQLException: ORA-01013: user
requested cancel of current operation" from Weblogic, and
"weblogic.transaction.internal.TimedOutException:" from my client.
Apparently, according to the original exception, an XA Resource has not
responded within 120 seconds, but through my testing, that XA resource isn't
necessarily a SQL connection waiting on a query.
Could someone provide insight as to what resource the system may be waiting
on? For in theory, to me, that's what is consuming our DB Pool. I'm assuming
that whatever they're waiting on (the DB apparently), they're ALL waiting on
it, and the reason that the pool spikes is because the system can never find
a "valid" connection, thus trying again with a new one. Is that what it
happening?
Clearly we basically have a tuning issue, but in fact, I'm looking for a
stability solution. When this occurred, our system run 3 load balanced
"legs". The middle leg was having the problem. When we dropped it, leg 3
picked up and then IT had the problem. However, once we killed all 3 of them
and restarted, they recovered.
So I'm looking for some idea to keep Weblogic alive, even at the expense of
booting off these offending transactions, and not having to restart the
servers.
Perhaps over time the system would have recovered on its own (most probably
in fact), but if we can get things cut off early (as well as beating users
that hit Refresh constantly when things "stick"), we can prevent these from
escalating the effectively downing the whole site.
Any thoughts would be appreciated.
Regards,
Will Hartung
([email protected])

Answer 1

Will Hartung wrote:
Weblogic 8.1, I don't recall which service pack, sp10? Maybe?Well, we're only up to sp5 here, so we'll have to catch up ;)
Please try this:
<JDBCConnectionPool CapacityIncrement="1"
DriverName="oracle.jdbc.driver.OracleDriver" InitialCapacity="45"
MaxCapacity="45" Name="OraclePool"
Password="*****"
Properties="user=******"
Targets="myserver" TestConnectionsOnCreate="true"
TestConnectionsOnRelease="false" TestConnectionsOnReserve="true"
KeepXAConnTillTxComplete="true"
TestTableName="dual" URL="jdbc:oracle:thin:@garrison.back:1521:IC"/>
<JDBCTxDataSource EnableTwoPhaseCommit="true"
JNDIName="dbpool" Name="OracleSource"
PoolName="OraclePool" Targets="myserver"/>
You don't want the pool in flux at runtime, either shrinking or
growing. Just have it make what you need at startup and keep it.
Test-on-release is a waste of cycles. Also, I set keep-xa-con
till tx complete, to ensure that only one connection is ever
used for a given tx. XA allows connections to be changed at the
granularity of a single JDBC call, but that provides no real
benefit, and some overhead.
Let me know if this helps, and do let me know what real
service pack you're running.
thanks
Joe
Anyway, here's the relevant stack trace. We had a strange WLS implosion last
night that we're tracking down. We saw these when our code was trying to get
a connection out of the datapool.
<Mar 1, 2006 10:09:12 PM PST> <Warning> <JTA> <BEA-110030> <XA resource
[weblogic.jdbc.wrapper.JTSXAResourceImpl] has not responded in the last 120
second(s).>
01 Mar 2006 22:09:21,652 WARN com.pfizer.ecms.as.MetricsSession - logEvent:
exception: start() failed on resource
'weblogic.jdbc.wrapper.JTSXAResourceImpl': XAER_RMFAIL : Resource manager is
unavailable
javax.transaction.xa.XAException: Internal error: XAResource
'weblogic.jdbc.wrapper.JTSXAResourceImpl' is unavailable
at
weblogic.transaction.internal.XAResourceDescriptor.checkResource(XAResourceD
escriptor.java:1019)
at
weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResour
ceDescriptor.java:572)
at
weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInf
o.java:1068)
at
weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceI
nfo.java:1001)
at
weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceIn
fo.java:203)
at
weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTra
nsactionImpl.java:419)
at weblogic.jdbc.jts.Driver.createLocalConnection(Driver.java:207)
at weblogic.jdbc.jts.Driver.connect(Driver.java:154)
at
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java
:298)
at
com.pfizer.ecms.asutil.DataAccessManager.getConnection(DataAccessManager.jav
a:117)
Here's our configuration:
<JDBCConnectionPool CapacityIncrement="1"
DriverName="oracle.jdbc.driver.OracleDriver" InitialCapacity="1"
MaxCapacity="150" Name="OraclePool"
Password="*****"
Properties="user=******" ShrinkPeriodMinutes="5"
Targets="myserver" TestConnectionsOnCreate="true"
TestConnectionsOnRelease="true" TestConnectionsOnReserve="true"
TestTableName="dual" URL="jdbc:oracle:thin:@garrison.back:1521:IC"/>
<JDBCTxDataSource EnableTwoPhaseCommit="true"
JNDIName="dbpool" Name="OracleSource"
PoolName="OraclePool" Targets="myserver"/>
Looking this up, I was pointed to a JTA FAQ page on edocs, and it basically
said we should be Refreshing the connection pool, but the RefreshMinutes
parameter is deprecated, and as you can see we're testing connections as is.
We're puzzled by a couple of things, almost certainly application related,
but still relevant. We have 150 connections allocated to the pool, but even
at peak load during the day we bounce around the 40 mark. When we got the
above exception, we were hitting the 150 ceiling. We don't quite know why
yet.
During this time, our DB server was pretty heaviliy loaded. There were some
horrendous queries running, some data imports, etc. Bascially a bunch of
stuff that happens at the beginning of the month. However, there weren't
necessarily a lot of actual users on the system, just a select few.
One of the things that was happening was a user was using the web interface,
and sent off a huge horrible query. When it didn't come back in a reasonable
amount of time, he simply hit "Refresh" on the page, thereby submitting the
query again (thank you Sir, may I have another!). This was partly why the
database was so heavily loaded.
Now, we're puzzled by a couple of things. One, is the spike of actual
connections in the DB Pool up to 150. This makes no sense.
Second, is this error detailed above. I've tried some contrived
experiements, and I can not raise this specific exception myself. My
experiment was to issue a "update table set column = 1 where id = 1" in a
generic SQL browser, thus locking the row, and then issue the identical
command through a Session Bean call. It has happily sat there for up to 10
minutes (600 seconds, which is our default <JTA Name="inf4.2"
TimeoutSeconds="600"/> setting), and it eventually times out, but not with
the same exception. Rather I get "java.sql.SQLException: ORA-01013: user
requested cancel of current operation" from Weblogic, and
"weblogic.transaction.internal.TimedOutException:" from my client.
Apparently, according to the original exception, an XA Resource has not
responded within 120 seconds, but through my testing, that XA resource isn't
necessarily a SQL connection waiting on a query.
Could someone provide insight as to what resource the system may be waiting
on? For in theory, to me, that's what is consuming our DB Pool. I'm assuming
that whatever they're waiting on (the DB apparently), they're ALL waiting on
it, and the reason that the pool spikes is because the system can never find
a "valid" connection, thus trying again with a new one. Is that what it
happening?
Clearly we basically have a tuning issue, but in fact, I'm looking for a
stability solution. When this occurred, our system run 3 load balanced
"legs". The middle leg was having the problem. When we dropped it, leg 3
picked up and then IT had the problem. However, once we killed all 3 of them
and restarted, they recovered.
So I'm looking for some idea to keep Weblogic alive, even at the expense of
booting off these offending transactions, and not having to restart the
servers.
Perhaps over time the system would have recovered on its own (most probably
in fact), but if we can get things cut off early (as well as beating users
that hit Refresh constantly when things "stick"), we can prevent these from
escalating the effectively downing the whole site.
Any thoughts would be appreciated.
Regards,
Will Hartung
([email protected])