XA & JMS javax.transaction.SystemException: SubCoordinator 'b00+127.0.0.1:7001+b+' not available

Question

Hi, I'm doing a "proof of concept" on XA transaction spanning two servers.
          In order to get it as simple as possible I have configured two servers and
          deployed the same MDB on both. I'm also only using JMS resources to simplify
          the test.
          The goal is to send a message to two different JMS servers (each deployed in
          a Weblogic server) both coordinated by an XA UserTransaction started in the
          client.
          The deployed MDB of each server just listens to a topic and prints out the
          received TextMessage.
          The configuration is as follows:
          * Server 1
          Domain name: a
          Server name: a00
          config.xml (extract):
          <JMSServer Name="a.a00.JMSServer" PagingStore="a.a00.JMSFileStore"
          Store="a.a00.JMSFileStore" Targets="a00">
          <JMSTopic JNDIName="xajms.test.Topic" Name="XA JMS Proof of Concept"
          StoreEnabled="true"/>
          </JMSServer>
          <JMSFileStore Directory="config/a/a00.jmsfilestore"
          Name="a.a00.JMSFileStore"/>
          <JMSConnectionFactory JNDIName="a.a00.JMSConnectionFactory"
          Name="a.a00.JMS Connection Factory" Targets="a00"
          UserTransactionsEnabled="true" XAConnectionFactoryEnabled="true"/>
          * Server 2
          Domain name: b
          Server name: b00
          config.xml (extract):
          <JMSServer Name="b.b00.JMSServer" PagingStore="b.b00.JMSFileStore"
          Store="b.b00.JMSFileStore" Targets="b00">
          <JMSTopic JNDIName="xajms.test.Topic" Name="XA JMS Proof of Concept"
          StoreEnabled="true"/>
          </JMSServer>
          <JMSFileStore Directory="config/b/b00.jmsfilestore"
          Name="b.b00.JMSFileStore"/>
          <JMSConnectionFactory JNDIName="b.b00.JMSConnectionFactory"
          Name="b.b00.JMS Connection Factory" Targets="b00"
          UserTransactionsEnabled="true" XAConnectionFactoryEnabled="true"/>
          and here is an extract of client code:
          068 UserTransaction utx;
          069
          070 InitialContext ctx1,ctx2;
          071
          072 System.out.println("Retrieving initial context for server 1");
          073 ctx1 = (InitialContext) getInitialContext(SERVER_1,PORT_1);
          074
          075 System.out.println("Retrieving initial context for server 2");
          076 ctx2 = (InitialContext) getInitialContext(SERVER_2,PORT_2);
          077
          078 utx = (UserTransaction)
          ctx1.lookup("javax.transaction.UserTransaction");
          079 utx.setTransactionTimeout(30);
          080
          081 System.out.println("Begining transaction");
          082 utx.begin();
          083
          084 System.out.println("Sending message to server 1");
          085 ref.send(ctx1, JMS_FACTORY_1, args[0]);
          086
          087 System.out.println("Sending message to server 2");
          088 ref.send(ctx2, JMS_FACTORY_2, args[0]);
          089
          090 System.out.println("Ending transaction...");
          091 if (args[1].equals("commit"))
          092 utx.commit();
          093 else
          094 utx.rollback();
          end the result of executing it from a third machine:
          CASE I:
          Retrieving initial context for server 1
          Retrieving initial context for server 2
          Begining transaction
          Sending message to server 1
          Sending message to server 2
          Ending transaction...
          javax.transaction.SystemException: SubCoordinator 'b00+127.0.0.1:7001+b+'
          not available
          Start server side stack trace:
          javax.transaction.SystemException: SubCoordinator 'b00+127.0.0.1:7001+b+'
          not available
          at
          weblogic.transaction.internal.TransactionImpl.abort(TransactionImpl.java:965
          at
          weblogic.transaction.internal.ServerSCInfo.startPrepare(ServerSCInfo.java:20
          0)
          at
          weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTran
          sactionImpl.java:1619)
          at
          weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTra
          nsactionImpl.java:217)
          at
          weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransaction
          Impl.java:189)
          at
          weblogic.transaction.internal.CoordinatorImpl.commit(CoordinatorImpl.java:68
          at
          weblogic.transaction.internal.CoordinatorImpl_WLSkel.invoke(Unknown Source)
          at
          weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:360)
          at
          weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:329)
          at
          weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:2
          2)
          at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:140)
          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:121)
          End server side stack trace
          <<no stack trace available>>
          --------------- nested within: ------------------
          weblogic.transaction.RollbackException: SubCoordinator
          'b00+127.0.0.1:7001+b+' not available
          Start server side stack trace:
          javax.transaction.SystemException: SubCoordinator 'b00+127.0.0.1:7001+b+'
          not available
          at
          weblogic.transaction.internal.TransactionImpl.abort(TransactionImpl.java:965
          at
          weblogic.transaction.internal.ServerSCInfo.startPrepare(ServerSCInfo.java:20
          0)
          at
          weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTran
          sactionImpl.java:1619)
          at
          weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTra
          nsactionImpl.java:217)
          at
          weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransaction
          Impl.java:189)
          at
          weblogic.transaction.internal.CoordinatorImpl.commit(CoordinatorImpl.java:68
          at
          weblogic.transaction.internal.CoordinatorImpl_WLSkel.invoke(Unknown Source)
          at
          weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:360)
          at
          weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:329)
          at
          weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:2
          2)
          at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:140)
          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:121)
          --------------- nested within: ------------------
          weblogic.transaction.RollbackException: SubCoordinator
          'b00+127.0.0.1:7001+b+' not available - with nested exception:
          [javax.transaction.SystemException: SubCoordinator 'b00+127.0.0.1:7001+b+'
          not available]
          at
          weblogic.transaction.internal.TransactionImpl.throwRollbackException(Transac
          tionImpl.java:1524)
          at
          weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTra
          nsactionImpl.java:265)
          at
          weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransaction
          Impl.java:189)
          at
          weblogic.transaction.internal.CoordinatorImpl.commit(CoordinatorImpl.java:68
          at
          weblogic.transaction.internal.CoordinatorImpl_WLSkel.invoke(Unknown Source)
          at
          weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:360)
          at
          weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:329)
          at
          weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:2
          2)
          at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:140)
          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:121)
          End server side stack trace
          - with nested exception:
          [javax.transaction.SystemException: SubCoordinator 'b00+127.0.0.1:7001+b+'
          not available
          Start server side stack trace:
          javax.transaction.SystemException: SubCoordinator 'b00+127.0.0.1:7001+b+'
          not available
                  at
          weblogic.transaction.internal.TransactionImpl.abort(TransactionImpl.java:965
                  at
          weblogic.transaction.internal.ServerSCInfo.startPrepare(ServerSCInfo.java:20
          0)
                  at
          weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTran
          sactionImpl.java:1619)
                  at
          weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTra
          nsactionImpl.java:217)
                  at
          weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransaction
          Impl.java:189)
                  at
          weblogic.transaction.internal.CoordinatorImpl.commit(CoordinatorImpl.java:68
                  at
          weblogic.transaction.internal.CoordinatorImpl_WLSkel.invoke(Unknown Source)
                  at
          weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:360)
                  at
          weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:329)
                  at
          weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:2
          2)
                  at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:140)
                  at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:121)
          End  server side stack trace
          at
          weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.
          java:85)
          at
          weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:136)
          at weblogic.rmi.internal.ProxyStub.invoke(ProxyStub.java:35)
          at $Proxy3.commit(Unknown Source)
          at
          weblogic.transaction.internal.TransactionImpl.commit(TransactionImpl.java:29
          4)
          at
          weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManag
          erImpl.java:247)
          at
          weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManag
          erImpl.java:240)
          at xajms.client.Send2BothTopics.main(Send2BothTopics.java:110)
          real 1m6.530s
          user 0m1.380s
          sys 0m0.050s
          If I swap message sending order the output changes to:
          CASE II:
          Retrieving initial context for server 1
          Retrieving initial context for server 2
          Begining transaction
          Sending message to server 2
          Sending message to server 1
          Ending transaction...
          weblogic.transaction.internal.TimedOutException: Timed out
          tx=0:8c9a9deee02fe21e after 30 seconds
          Start server side stack trace:
          weblogic.transaction.internal.TimedOutException: Timed out
          tx=0:8c9a9deee02fe21e after 30 seconds
          at
          weblogic.transaction.internal.ServerTransactionImpl.wakeUp(ServerTransaction
          Impl.java:1228)
          at
          weblogic.transaction.internal.ServerTransactionManagerImpl.processTimedOutTr
          ansactions(ServerTransactionManagerImpl.java:488)
          at
          weblogic.transaction.internal.TransactionManagerImpl.wakeUp(TransactionManag
          erImpl.java:1629)
          at
          weblogic.transaction.internal.ServerTransactionManagerImpl.wakeUp(ServerTran
          sactionManagerImpl.java:451)
          at
          weblogic.transaction.internal.TransactionManagerImpl$1.run(TransactionManage
          rImpl.java:1595)
          at java.lang.Thread.run(Thread.java:479)
          End server side stack trace
          <<no stack trace available>>
          --------------- nested within: ------------------
          weblogic.transaction.RollbackException: Timed out tx=0:8c9a9deee02fe21e
          after 30 seconds
          Start server side stack trace:
          weblogic.transaction.internal.TimedOutException: Timed out
          tx=0:8c9a9deee02fe21e after 30 seconds
          at
          weblogic.transaction.internal.ServerTransactionImpl.wakeUp(ServerTransaction
          Impl.java:1228)
          at
          weblogic.transaction.internal.ServerTransactionManagerImpl.processTimedOutTr
          ansactions(ServerTransactionManagerImpl.java:488)
          at
          weblogic.transaction.internal.TransactionManagerImpl.wakeUp(TransactionManag
          erImpl.java:1629)
          at
          weblogic.transaction.internal.ServerTransactionManagerImpl.wakeUp(ServerTran
          sactionManagerImpl.java:451)
          at
          weblogic.transaction.internal.TransactionManagerImpl$1.run(TransactionManage
          rImpl.java:1595)
          at java.lang.Thread.run(Thread.java:479)
          --------------- nested within: ------------------
          weblogic.transaction.RollbackException: Timed out tx=0:8c9a9deee02fe21e
          after 30 seconds - with nested exception:
          [weblogic.transaction.internal.TimedOutException: Timed out
          tx=0:8c9a9deee02fe21e after 30 seconds]
          at
          weblogic.transaction.internal.TransactionImpl.throwRollbackException(Transac
          tionImpl.java:1524)
          at
          weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTra
          nsactionImpl.java:265)
          at
          weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransaction
          Impl.java:189)
          at
          weblogic.transaction.internal.CoordinatorImpl.commit(CoordinatorImpl.java:68
          at
          weblogic.transaction.internal.CoordinatorImpl_WLSkel.invoke(Unknown Source)
          at
          weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:360)
          at
          weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:329)
          at
          weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:2
          2)
          at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:140)
          at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:121)
          End server side stack trace
          - with nested exception:
          [weblogic.transaction.internal.TimedOutException: Timed out
          tx=0:8c9a9deee02fe21e after 30 seconds
          Start server side stack trace:
          weblogic.transaction.internal.TimedOutException: Timed out
          tx=0:8c9a9deee02fe21e after 30 seconds
                  at
          weblogic.transaction.internal.ServerTransactionImpl.wakeUp(ServerTransaction
          Impl.java:1228)
                  at
          weblogic.transaction.internal.ServerTransactionManagerImpl.processTimedOutTr
          ansactions(ServerTransactionManagerImpl.java:488)
                  at
          weblogic.transaction.internal.TransactionManagerImpl.wakeUp(TransactionManag
          erImpl.java:1629)
                  at
          weblogic.transaction.internal.ServerTransactionManagerImpl.wakeUp(ServerTran
          sactionManagerImpl.java:451)
                  at
          weblogic.transaction.internal.TransactionManagerImpl$1.run(TransactionManage
          rImpl.java:1595)
                  at java.lang.Thread.run(Thread.java:479)
          End  server side stack trace
          at
          weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.
          java:85)
          at
          weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:136)
          at weblogic.rmi.internal.ProxyStub.invoke(ProxyStub.java:35)
          at $Proxy3.commit(Unknown Source)
          at
          weblogic.transaction.internal.TransactionImpl.commit(TransactionImpl.java:29
          4)
          at
          weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManag
          erImpl.java:247)
          at
          weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManag
          erImpl.java:240)
          at xajms.client.Send2BothTopics.main(Send2BothTopics.java:110)
          real 1m32.231s
          user 0m1.390s
          sys 0m0.030s
          not to be said that I have "some" questions about it:
          1. Why it doesn't work?
          2. Why order of message sending brings two different results?
          3. In CASE I:
          What is the meaning of the exception "javax.transaction.SystemException:
          SubCoordinator 'b00+127.0.0.1:7001+b+' not available"
          Why is it using the loopback (127.0.0.1) ip? Is that code being runned
          inside the server with the SERVER_2 (192.168.1.3) ip?
          4. In case II:
          How is that the message of the exception is
          "weblogic.transaction.internal.TimedOutException: Timed out
          tx=0:8c9a9deee02fe21e after 30 seconds" when it's clear the program took
          more than one and a half minutes to execute?
          Most of the time is spent after printing the "Ending transaction..." line.
          The first part of the program is executed in a few seconds (about 2 or 3 it
          depends of the time to get initial context against both servers)
          I'm really puzzled with this. I don't know what to think about it. Is this
          so weird nobody have tried something like this?
          I have done some other test for which the results are:
          a) Commenting lines related to UserTransaction (82,91-94)
          It works OK. Both servers receive the message.
          b) Commenting lines 85 or 88 (related to sending messages to each server)
          It works OK. Both servers receive the message. I pressume there is no
          "real" XA transaction as there is only one resource involved on it.
          c) In CASE I launching the client with "rollback" as second parameter I
          neither get any exception nor messages are delivered which would be OK
          d) In CASE II launching the client with "rollback" as second parameter I get
          a "javax.transaction.SystemException: Timeout during rollback processing"
          and messages are not delivered.
          Any idea?. I really need this to work in order to free our server for heavy
          asyncronous loads and deploy new stuff on it.
          PS.
          I attach both servers config.xml files and both MDB and client full code. If
          you are going to test it you must add a new user and group to weblogic
          default file realm:
          New user: subscriber
          New group: xajms
          of course you must add the new user to the new group in order to make it
          work.
          Here are the lines extracted from server's domain "fileRealm.properties"
          file reflecting the new user and group (defined on ejb-jar.xml and
          weblogic-ejb-jar.xml files on the "role" section):
          group.xajms=subscriber
          user.subscriber=0x6b9a704bebc709dd083edd61ed000236eb23987f
          The client receives two parameters. The first one is a string to be sent as
          message body, the second one is either "commit" or "rollback" just to test
          XA transaction doing it's work as expected.
          Thanks in advance.
          Regards.
          Ignacio.
          [xajmspoc.zip]
          

Answer 1

do you connect to 10g database inside onBusinessEvent using Non-XA/XA connection??. I guess it could be bcoz of Non-XA connection (or) XA Without Global Transaction support, and this connection is listed with the transaction manager and it is trying to commit it after onMessage.
For Global Transactions options, go to your weblogic console and check the following path.
(Datasource -> Advanced Options -> Global Transactions -> Honor Global Transactions should be true)
Let me know if you need any help
Rao Kotha.