AQ Callbacks - Blocking background processes and best practices.

Question

We are running several queues within our company and one of them uses pl/sql callback functionnality.
Basically, several triggers can enqueue a message when underlying tables are updated. The goal of the callback is to "treat" those messages, which means dequeuing the messages and passing them to some procedure (determined through a confguration table and the type of message). I don't know if i'm clear enough on this but it is as important.
In general the mechanism works perfectly but we noticed in one of our databases that, after a relatively big amount of messages enqueuing (~ 500-1000 in one trasanction), there are numerous background processes blocked on system table SYS.AQ_SRVNTFN_TABLEI. In fact, the queue starts growing and messages are not dequeued anymore by the callback, that doesn't seem to be executed anymore at all.
We actually were also unable to re-compile the package that holds the callback procedure nor removing/adding the reference of the subscriber to the queue with DBMS_AQ. It would hang there forever...
We are running Oracle Database 10g Enterprise Edition Release *10.2.0.4.0* and the value of AQ_TM_PROCESSES = 0.
I don't necesseraly have a complete and clear view of how the queue mechanism works behind the scenes so forgive for some foolish things I could say :-)
Diagnosis:
After some research, it seems the backgrounds processes are blocked on the system table SYS.AQ_SRVNTFN_TABLE on some index for MSGID.
If I understand correctly how the system works, the callback gets executed for a specific MSGID as we use it in the callback procedure to dequeue this message.
I've also discovered that the default value for the WAIT dequeue option is FOREVER...
So my idea was that, for some reason, the callback tries to dequeue a message that does not exist in the queue and... waits forever for the message to "appear".
This at first seemed pretty unlikely to me: why would the callback be executed if no message with provided MSGID exists... then you start doubting :-)
Attempt to resolve:
We've decided to alter the WAIT option to not let the dequeue in the callback wait forever.
We have made some tests (outside the callback) with NO_WAIT and also with a wait of a few seconds. Both solution prooved right so we added a wait of 60s in the callback and added some tracing.
Since then, no more background processes seems to hang there and we are able to alter the callback procedure normally. But with the added tracing, we noticed an unexplained behavior with the execution of the callback:
- callback runs
- ORA-25263: no message in queue QUEUE_OWNER.MULTISRC_NOTIFQ with message ID B4FFD1115523A46EE040007F0100304F
- callback runs again and message is dequeued and treated correctly
The first oracle error happens fast, not after 60s as the WAIT option specifies.
Questions:
<li>Is the way the callback method is implemented correct (see code below) ?</li>
<li>Do we need to commit in the callback method ? What implies committing or not ?</li>
<li>How wouold you explain the behavior of the callback after setting the dequeue WAIT to 60s and why it does not actually wait for 60s ?</li>
<>
<>
<>
<>
The configuration of the queue and the callback is as follow:
- the type of the payload is SYS.ANYDATA
- an extra index is create on CORRID column
begin
   -- create queue table
   begin
      DBMS_AQADM.CREATE_QUEUE_TABLE(QUEUE_TABLE        => 'QUEUE_OWNER.MULTISRC_NOTIFTAB'
                                   ,QUEUE_PAYLOAD_TYPE => 'SYS.ANYDATA'
                                   ,MULTIPLE_CONSUMERS => true);
   end;
   -- create and start queue
   begin DBMS_AQADM.CREATE_QUEUE(QUEUE_NAME => 'QUEUE_OWNER.MULTISRC_NOTIFQ', QUEUE_TABLE => 'QUEUE_OWNER.MULTISRC_NOTIFTAB'); end;
   begin DBMS_AQADM.START_QUEUE(QUEUE_NAME => 'QUEUE_OWNER.MULTISRC_NOTIFQ'); end;
   -- grant access to the queue to PDO
   -- add a subscriber to the queue and register the plsql to execute
   begin
      DBMS_AQADM.ADD_SUBSCRIBER(QUEUE_NAME => 'QUEUE_OWNER.MULTISRC_NOTIFQ'
                               ,SUBSCRIBER => SYS.AQ$_AGENT('MULTISRC_NOTIFSUBSCR', null, null));
      DBMS_AQ.REGISTER(SYS.AQ$_REG_INFO_LIST(SYS.AQ$_REG_INFO('QUEUE_OWNER.MULTISRC_NOTIFQ:MULTISRC_NOTIFSUBSCR'
                                                             ,DBMS_AQ.NAMESPACE_AQ
                                                             ,'plsql://PDO.PO_NOTIFY.MULTISRC_NOTIF_SUBSCRIBER?PR=0'
                                                             ,HEXTORAW('FF')))
                      ,1);
   end;
end;
create index queue_owner.multisrcq_corrid on queue_owner.multisrc_notiftab (CORRID)
The callback procedure is as follow:
procedure MULTISRC_NOTIF_SUBSCRIBER(context  raw,
                                      REGINFO  SYS.AQ$_REG_INFO,
                                      DESCR    SYS.AQ$_DESCRIPTOR,
                                      PAYLOAD  raw,
                                      PAYLOADL number) is
    L_METHOD constant varchar2(50) := 'MULTISRC_NOTIF_SUBSCRIBER';
    R_DEQUEUE_OPTIONS    DBMS_AQ.DEQUEUE_OPTIONS_T;
    R_MESSAGE_PROPERTIES DBMS_AQ.MESSAGE_PROPERTIES_T;
    V_MESSAGE_HANDLE     raw(16);
    O_PAYLOAD            ANYDATA;
    cursor C_TREATMENTS(P_ENTITY in varchar2) is
      select T.MNOT_PROCEDURE_C
        from TA_GEN.MULTISRC_NOTIF_TREATMENTS T
       where T.MNOT_ENTITY_C = P_ENTITY
         and T.MNOT_BEGIN_D < sysdate
         and ((T.MNOT_END_D > sysdate and T.MNOT_END_D is not null) or
             (T.MNOT_END_D is null))
       order by T.MNOT_PRIORITY_N asc;
    WK_CORRID    varchar2(128);
    WK_ENTITY    TA_GEN.MULTISRC_NOTIF_TREATMENTS.MNOT_ENTITY_C%type;
    WK_PROCEDURE TA_GEN.MULTISRC_NOTIF_TREATMENTS.MNOT_PROCEDURE_C%type;
    CT_EXEC number(2) := 0;
  begin
    -- DGH: 15.12.11 / added a wait of 60 seconds for dequeue to avoid infinite waiting (default is FOREVER)
    R_DEQUEUE_OPTIONS.WAIT := 60;
    -- dequeue message
    R_DEQUEUE_OPTIONS.MSGID         := DESCR.MSG_ID;
    R_DEQUEUE_OPTIONS.CONSUMER_NAME := DESCR.CONSUMER_NAME;
    DBMS_AQ.DEQUEUE(QUEUE_NAME         => DESCR.QUEUE_NAME,
                    DEQUEUE_OPTIONS    => R_DEQUEUE_OPTIONS,
                    MESSAGE_PROPERTIES => R_MESSAGE_PROPERTIES,
                    PAYLOAD            => O_PAYLOAD,
                    MSGID              => V_MESSAGE_HANDLE);
    -- extract entity name
    WK_CORRID := R_MESSAGE_PROPERTIES.CORRELATION;
    WK_ENTITY := SUBSTR(WK_CORRID,
                        INSTR(WK_CORRID, '##') + 2,
                        (INSTR(WK_CORRID, '##', 1, 2) -
                        INSTR(WK_CORRID, '##')) - 2);
    -- execute treatment(s)
    open C_TREATMENTS(WK_ENTITY);
    loop
      fetch C_TREATMENTS
        into WK_PROCEDURE;
      exit when C_TREATMENTS%notfound;
      execute immediate 'begin ' || WK_PROCEDURE || '(:MSG); end;'
        using O_PAYLOAD;
      CT_EXEC := CT_EXEC + 1;
    end loop;
    close C_TREATMENTS;
  exception
    when others then
      if C_TREATMENTS%isopen then close C_TREATMENTS; end if;
      PO_NOTIFY.TRACE_MULTISRC_NOTIF(L_METHOD,
                                     sqlerrm,
                                     DESCR.MSG_ID,
                                     R_MESSAGE_PROPERTIES.CORRELATION,
                                     WK_ENTITY,
                                     WK_PROCEDURE);
      rollback;
  end MULTISRC_NOTIF_SUBSCRIBER;

Answer 1

Helping you with the specific issue is going to be difficult without direct access to the servers but given the importance this system seems to have to your business why are you not running on a fully supported version (10.2 has been in extended support for more than 6 months) and even in the current configuration not patched to 10.2.0.5?
My instinct would be to focus on moving to 11.2.0.3 as quickly as possible with a corresponding change to a current operating system version if your O/S is similarly out of date.