XA overhead in call to prepare, taking up to 1000 ms

Question

Hello everyone.
In a particular use-case on our load-test environment (similar to production) where customer data is being updated via a SOAP from a weblogic 10.3 (JDBC driver 11.2.0.2.0) to two 11gR2 RAC cluster (which leads to a lot of SQL queries, including DML and a JMS message) we experience execution times of oracle.jdbc.xa.client.OracleXAResource.prepare(Xid) (which is being called once at the end of the service-call) which are far from being acceptable, about 300-1000 ms.
We measured the execution times with java profilers (dynaTrace, MissionControl). To ensure these values are valid we put the ojdbc6_g.jar in place and saw the long times in the logs.
Example:
<record>
<date>2011-07-27T16:48:45</date>
<millis>1311785325858</millis>
<sequence>7265</sequence>
<logger>oracle.jdbc.xa.client</logger>
<level>FINE</level>
<class>oracle.jdbc.xa.client.OracleXAResource</class>
<method>prepare</method>
<thread>11</thread>
<message>41B70007 Exit [354.443ms]</message>
</record>
We took a TCP dump in order to see what is being sent to the database but couldn't decompile what exactly is being transfered via the NET8 protocol.
From what I've read (http://download.oracle.com/docs/cd/B28359_01/java.111/b31224/xadistra.htm) the thin driver should be using the native XA by default so this should not be a reason for the poor performance.
We have many other services that do similar DML but don't show this behavior, so it must be something specific.
From the profiling and TCP dumping we are pretty sure the time is being spent on the DB side.
This assumption was strengthened by the odd fact that this monday, after no usage of the system on the weekend, the overhead suddenly just dissapeared! The execution times were as low as one would expect (~5-10 ms). We saw an out of memory (ORA-4030) occured on Saturday, which is still in investigation by the provider.
I suspected that the long prepare times would come back after some load, so I initiated a load test which executes these use cases and simulates a real life scenario. After 1 or 2 hours, this was the case. Now we are in the same situation as before. Again it is reproducable with single calls and no other load on the DB. I imagined there might have been restarts of the instances or something similar in order to recover from the ORA-4030 so I initiated restarts of all instances but without success.
This is were we are right now, the experiences so far lead imho to the following conclusions/assumptions:
1. The time is being spent on the DB (maybe partly somewhere in the network)
2. We are most probably experiencing an erroneous behavior because we had a situation were the issue did not occur, but we don't know why (yet)
3. Maybe it was by accidential circumstances on monday that the problem dissapeared and it had nothing to do with our load-test later on that it is back now (since the physical hardware (DB server and storage) is shared but we see no contention on CPU,RAM or I/O)
4. JMS should not be the issue because we see a dedicated prepare call which is fast and it's handled locally on the AppServer
The big question is, how can we pin down where exactly on the DB the time is being spent? Is there a way to find out how long each participating RM takes in order to handle the prepare-call?
Any help would be greatly appreciated, these execution times can threaten our SLA.
Kind regards,
Thomas
PS: We've opened an SR as well, but there's has not been a lot of useful information so far. This statement is not very promising "There is no specific mechanism to find out why the prepare state takes time."

Answer 1

Hi Thomas, you can do some test before recommend enable XA at RAC level.
(Please check if the jdbc driver need access to PL/SQL level of XA procedures or the JDBC just use the API XA native of Oracle 11).
check if you are using the jdbc for 11G.
- A simple test to check the response time just please do shutdown abort (one node) and check the response time on the other node.
- After do this test, shutdown the database and start the databases (both) to start a clean scenario and do some tests, if you feel the system goes slow just check the lock at RAC level if you see the same SID locking the same object on both nodes, well you need to run XA scripts on ypu database, if not keep looking. if you don't have the script to check the lock at RAC level, just let me know I can publish the scripts for you. on RAC 10G all the time just I run the XA scripts because some client need some PL/SQL api XA. EX: .COM or .NET over windows 2003 or windows 2008.