Overview of the innards of the AQ process?

Question

We came up with the brilliant idea to use AQ to create a system like the log4j package for Java - in short a system which we can use to log all sorts of messages (debug info, warnings, errors). The system allows for multiple message consumers, and each consumer has a rule determining which messages it is interested in. At the moment, we have one consumer which simply writes the messages to a log table, but in the future we also want to add consumers that send emails in case of an error, etc. The sky is the limit!
Alas. We have it working without a problem on a test database, which is a Windows server, but on our production database the whole thing grinds to a squeaking halt after a while. The production database, version 10.1.0.5, runs on an Apple XServe.
So, what happens? After a while, we see that the table SYS.AQ_SRVNTFN_TABLE fills up with records, which are not being removed. Also, the table AQ$LOGMESSAGE_QTAB_H and AQ$LOGMESSAGE_QTAB_I fill up; no new messages are being written to the final log table (F_SYST_LOGMESSAGE). It seems that the entire AQ process halts. The only way we found to get it going again, is to remove all records from AQ_SRVNTFN_TABLE and recreate the queue and queue table.
We are unable to figure out what goes wrong. This is partly caused by our inexperience with AQ. Is there a clear description of the entire AQ process, from enqueuing a message, to sending it to the right queue, to dequeuing the message, complete with tables and views and server processes involved? What would be the best way to find out why our AQ system breaks down? Has anyone experience with AQ on Apple?

Answer 1

We are already using a commercial product; it is called Oracle Advanced Queuing. We have it running without problems on our development system. We need to figure out why the whole Advanced Queuing mechanism fails in our production environment. What would be really helpful is an overview of the whole AQ process, including tables and actions on those tables. For instance, the AQ documentation states that messages are enqueued and dequeued by Jxxx processes - but what exactly do those processes do? Where do they read their data from, and where do they write their data to? What's the role of AQ_SRVNTFN_TABLE? Why do messages 'get stuck' there?
Some more background information:
- no errors are generated in the alert log or the trace files
- we create the queue, queue table and payload as follows:
CREATE OR REPLACE TYPE TLogMessage AS OBJECT
sequence_nr NUMBER,
creationtime TIMESTAMP,
module VARCHAR2(4000), -- the module that generated the message
severity NUMBER, -- severity
message VARCHAR2(4000), -- the log message
userid VARCHAR2(4000), -- the user for which the log message applies
additional_info CLOB -- additional info
BEGIN
DBMS_AQADM.CREATE_QUEUE_TABLE(
queue_table => 'MTDX.LOGMESSAGE_QTAB',
multiple_consumers => TRUE,
queue_payload_type => 'MTDX.TLogMessage',
storage_clause => 'TABLESPACE DOCUMENT'
DBMS_AQADM.CREATE_QUEUE(
queue_name => 'logmessage_queue',
queue_table => 'MTDX.LOGMESSAGE_QTAB'
DBMS_AQADM.START_QUEUE(
queue_name => 'logmessage_queue'
END;
The following procedure is used to enqueue the log messages:
     PROCEDURE Log(
          pModule               VARCHAR2,
          pSeverity          NUMBER,
          pMessage          VARCHAR2,
          pUserId               VARCHAR2 DEFAULT NULL,
          pAdditionalInfo     CLOB DEFAULT NULL
     IS
          vMessage               TLogMessage;
          vEnqueueOptions          DBMS_AQ.ENQUEUE_OPTIONS_T;
          vMsgProperties          DBMS_AQ.MESSAGE_PROPERTIES_T;
          vMessageHandle          RAW(16);
          vSequenceNr               NUMBER;
     BEGIN
          SELECT SYLMSEQ.NEXTVAL
          INTO vSequenceNr
          FROM DUAL;
          vMessage := TLogMessage(
                              sequence_nr => vSequenceNr,
                              creationtime => SYSTIMESTAMP,
                              module => pModule,
                              severity => pSeverity,
                              message => pMessage,
                              userid => pUserId,
                              additional_info => pAdditionalInfo
          vEnqueueOptions.visibility := DBMS_AQ.IMMEDIATE;
          vMsgProperties.correlation := pModule;
          vMsgProperties.priority := -pSeverity;
          -- Enqueue the message to all subscribers
          DBMS_AQ.ENQUEUE(
               queue_name               => QUEUE_NAME,
               enqueue_options          => vEnqueueOptions,
               message_properties     => vMsgProperties,
               payload                    => vMessage,
               msgid                    => vMessageHandle
     EXCEPTION
          WHEN no_subscribers THEN
               -- No subscribers on the queue; ignore
               NULL;
     END;
And finally the following callback procedure is used by the queue consumer to write messages to the log table:
     PROCEDURE TableLoggerCallback(
          context          RAW,
          reginfo          SYS.AQ$_REG_INFO,
          descr          SYS.AQ$_DESCRIPTOR,
          payload          RAW,
          payloadl     NUMBER
     IS
          vDequeueOptions          DBMS_AQ.DEQUEUE_OPTIONS_T;
          vMsgProperties          DBMS_AQ.MESSAGE_PROPERTIES_T;
          vMessageHandle          RAW(16);
          vMessage               TLogMessage;
     BEGIN
          vDequeueOptions.msgid := descr.msg_id;
          vDequeueOptions.consumer_name := descr.consumer_name;
          DBMS_AQ.DEQUEUE(
               queue_name                => descr.queue_name,
               dequeue_options          => vDequeueOptions,
               message_properties     => vMsgProperties,
               payload                    => vMessage,
               msgid                    => vMessageHandle
          INSERT INTO F_SYST_LOGMESSAGE
               SYLM_SEQUENCE,
               SYLM_TIMESTAMP,
               SYLM_MODULE,
               SYLM_SEVERITY,
               SYLM_MESSAGE,
               SYLM_LOGGER,
               SYLM_USERID,
               SYLM_ADDITIONALINFO
          VALUES
               vMessage.sequence_nr,
               vMessage.creationtime,
               vMessage.module,
               vMessage.severity,
               vMessage.message,
               descr.consumer_name,
               vMessage.userid,
               vMessage.additional_info
          COMMIT;
     END;
-----