Apply slowdown on transaction with ROLLBACK TO UBA LCRs

Question

I have streams configuration consisting of several captures and propagations on four servers an one destination db with several applys.
Each capture has its own propagation and its own apply on destination server.
All works almost fine.
But sometimes one apply process becomes slowly.
As each stream has heartbeat table with 30 seconds insert/delete sequence, normal delay between message capture time and apply time 1-3 minutes as max.
But sometimes it became 3-4 hours.
I started to search the root of problem and found following:
When problem exist, apply reader process in latch shared pool contention. The OS process of apply reader is at 100% CPU usage.
DBA_APPLY_SPILL_TXN showing one transaction for that apply.
I use following script to see contents of that transaction:
DECLARE
  TYPENM   VARCHAR2(61);
  DDLLCR   SYS.LCR$_DDL_RECORD;
  PROCLCR  SYS.LCR$_PROCEDURE_RECORD;
  ROWLCR   SYS.LCR$_ROW_RECORD;
  RES      NUMBER;
  NEWLIST  SYS.LCR$_ROW_LIST;
  OLDLIST  SYS.LCR$_ROW_LIST;
  DDL_TEXT CLOB;
  EXT_ATTR ANYDATA;
  I        NUMBER;
BEGIN
  I := 0;
  SELECT COUNT(*)
    INTO I
    FROM aq$qta_strm1 T
   WHERE T.QUEUE = 'QA_STRM1'
   ORDER BY T.MSG_ID ASC;
  DBMS_OUTPUT.PUT_LINE('### CNT: ' || I);
  I := 0;
  FOR C IN (SELECT *
              FROM aq$qta_strm1 T
             WHERE T.QUEUE = 'QA_STRM1'
             ORDER BY T.MSG_ID ASC) LOOP
    IF (C.USER_DATA IS NOT NULL) THEN
      TYPENM := C.USER_DATA.GETTYPENAME();
      IF (TYPENM = 'SYS.LCR$_ROW_RECORD') THEN
        RES := C.USER_DATA.GETOBJECT(ROWLCR);
          DBMS_OUTPUT.PUT_LINE('MSG_ID: ' || C.MSG_ID);
          PRINT_LCR(C.USER_DATA);
          I := I + 1;
      END IF;
    END IF;
  END LOOP;
  DBMS_OUTPUT.PUT_LINE('### NN: ' || I);
END;And it gives me following:
<tt>
### CNT: 3150
MSG_ID: 03690459
source database: SOURCEDB.MYDOMAIN.COM
owner:
object:
is tag null: Y
command_type: ROLLBACK TO UBA
transaction_id: 9.46.1687138
MSG_ID: 03690460
source database: SOURCEDB.MYDOMAIN.COM
owner:
object:
is tag null: Y
command_type: ROLLBACK TO UBA
transaction_id: 9.46.1687138
MSG_ID: 03690461
source database: SOURCEDB.MYDOMAIN.COM
owner:
object:
is tag null: Y
command_type: ROLLBACK TO UBA
transaction_id: 9.46.1687138
MSG_ID: 03690462
source database: SOURCEDB.MYDOMAIN.COM
owner:
object:
is tag null: Y
command_type: ROLLBACK TO UBA
transaction_id: 9.46.1687138
+... and so on ...+
</tt>
I didn't find any description about lcr type ROLLBACK TO UBA.
Whole transaction consists of messages of such type.
So I just put transaction to IGNORE_TRANSACTION parameter of apply and problem disappeared for now.
But I believe it will come back again.
So, who can explain me what lcr type ROLLBACK TO UBA means? where it come from? and how to avoid this problem again?
PS:
Source database 10.2.0.4 64bit-linux
Dest. db 11.2.0.3 64bit-linux
Edited by: user5464827 on 02.04.2013 0:05

Answer 1

Hi Onno,
If you have enabled "auto commit", then you cannot call methods like "commit()" or "rollback()". After you have executed your INSERT, a "commit" automatically happens -- if the INSERT succeeds -- or a "rollback" automatically happens (if the INSERT fails). You are calling the "commit()" method when there is no open transaction -- which is wrong and therefore you get an error (SQLException).
Same problem occurs when you catch the error caused by your "commit" and try to "rollback" -- you cannot "rollback" when there is no open transaction.
As I see it, you have two choices:
1. disable auto commit, or
2. remove the calls to "commit()" and "rollback()"
Hope this helps you.
Good Luck,
Avi.