Performance issues when using AQ notification with one consumer

Question

We have developed a system to load data from a reservation database to a reporting database
A a certain point in the proces, a message with the identifier of the reservation is enqueued to a queue (multi-consumer) on the same DB and then propagated to a similar queue on the REP database.
This queue (multi-consumer) has AQ notification enabled (with one consumer) which calls the queue_callback procedure which
- dequeues the message
- calls a procedure to load the Resv data into the Reporting schema (through DB link)
We need each message to be processed ONLY ONCE thus the usage of one single subscriber (consumer)
But when load testing our application with multiple threads, the number of records created in the Reservation Database becomes quite large, meaning a large number of messages going through the first queue and propagating to the second queue (very quickly).
But messages are not processed fast enough by the 2nd queue (notification) which falls behind.
I would like to keep using notification as processing is automatic (no need to set up dbms_jobs to dequeue etc..) or something similar
So having read articles, I feel I need to use:
- multiple subscribers to the 2nd queue where each message is processed only by one subscriber (using a rule : say 10 subscribers S0 to S10 with Si processing messages where last number of the identifier is i )
problem with this is that there is an attempt to process the message for each subscriber, isn't there
- a different dequeuing method where many processes are used in parallel , with each message is processed only by one subscriber
Does anyone have experience and recommendations to make on how to improve throughput of messages?
Rgds
Philippe

Answer 1

Hi, thanks for your interest
I am working with 10.2.0.4
My objective is to load a subset of the reservation data from the tables in the first DB (Reservation-OLTP-150 tables)
to the tables in the second DB (Reporting - about 15 tables at the moment), without affecting performance on the Reservation DB.
Thus the choice of advanced queueing (asyncronous )
- I have 2 similar queues in 2 separate databases ( AND Reporting)
The message payload is the same on both (the identifier of the reservation)
When a certain event happens on the RESERVATION database, I enqueue a message on the first database
Propagation moves the same message data to the second queue.
And there I have notification sending the message to a single consumer, which:
- calls dequeue
- and the data load procedure, which load this reservation
My performance difficulties start at the notification but I will post all the relevant code before notification, in case it has an impact.
- The 2nd queue was created with a script containing the following (similar script for fisrt queue)
dbms_aqadm.create_queue_table( queue_table => '&&CQT_QUEUE_TABLE_NAME',
queue_payload_type => 'RESV_DETAIL',
comment => 'Report queue table',
multiple_consumers => TRUE,
message_grouping => DBMS_AQADM.NONE,
compatible => '10.0.0',
sort_list => 'ENQ_TIME',
primary_instance => '0',
secondary_instance => '0');
dbms_aqadm.create_queue (
queue_name => '&&CRQ_QUEUE_NAME',
queue_table => '&&CRQ_QUEUE_TABLE_NAME',
max_retries => 5);
- ENQUEUING on the first queue (snippet of code)
o_resv_detail DLEX_AQ_ADMIN.RESV_DETAIL;
o_resv_detail:= DLEX_AQ_ADMIN.RESV_DETAIL(resvcode, resvhistorysequence);
DLEX_RESVEVENT_AQ.enqueue_one_message (o_resv_detail);
where DLEX_RESVEVENT_AQ.enqueue_one_message is :
PROCEDURE enqueue_one_message (msg IN RESV_DETAIL)
IS
enqopt           DBMS_AQ.enqueue_options_t;
mprop           DBMS_AQ.message_properties_t;
enq_msgid           dlex_resvevent_aq_admin.msgid_t;
BEGIN
DBMS_AQ.enqueue (queue_name => dlex_resvevent_aq_admin.c_resvevent_queue,
enqueue_options => enqopt,
message_properties => mprop,
payload => msg,
msgid => enq_msgid
END;
- PROPAGATION: The message is dequeued from 1st queue and enqueued automatically by AQ propagation into this 2nd queue.
(using a call to the following 'wrapper' procedure)
PROCEDURE schedule_propagate (
src_queue_name IN VARCHAR2,
destination IN VARCHAR2 DEFAULT NULL
IS
sprocname dlex_types.procname_t:= 'dlex_resvevent_aq_admin.schedule_propagate';
BEGIN
DBMS_AQADM.SCHEDULE_PROPAGATION(queue_name => src_queue_name,
                                        destination => destination,
latency => 10);
EXCEPTION
WHEN OTHERS
THEN
DBMS_OUTPUT.put_line (SQLERRM || ' occurred in ' || sprocname);
END schedule_propagate;
- For 'NOTIFICATION': ONE subscriber was created using:
EXECUTE DLEX_REPORT_AQ_ADMIN.add_subscriber('&&STQ_QUEUE_NAME','&&STQ_SUBSCRIBER',NULL,NULL, NULL);
this is a wrapper procedure that uses:
DBMS_AQADM.add_subscriber (queue_name => p_queue_name, subscriber => subscriber_agent );
Then notification is registered with:
EXECUTE dlex_report_aq_admin.register_notification_action ('&&AQ_SCHEMA','&&REPORT_QUEUE_NAME','&&REPORT_QUEUE_SUBSCRIBER');
- job_queue_processes is set to 10
- The callback procedure is as follows
CREATE OR REPLACE PROCEDURE DLEX_AQ_ADMIN.queue_callback
context RAW,
reginfo SYS.AQ$_REG_INFO,
descr SYS.AQ$_DESCRIPTOR,
payload RAW,
payloadl NUMBER
IS
s_procname CONSTANT VARCHAR2 (40) := UPPER ('queue_callback');
r_dequeue_options DBMS_AQ.DEQUEUE_OPTIONS_T;
r_message_properties DBMS_AQ.MESSAGE_PROPERTIES_T;
v_message_handle RAW(16);
o_payload RESV_DETAIL;
BEGIN
r_dequeue_options.msgid := descr.msg_id;
r_dequeue_options.consumer_name := descr.consumer_name;
DBMS_AQ.DEQUEUE(
queue_name => descr.queue_name,
dequeue_options => r_dequeue_options,
message_properties => r_message_properties,
payload => o_payload,
msgid => v_message_handle
-- Call procedure to load data from reservation database to Reporting DB through the DB link
dlex_report.dlex_data_load.load_reservation
( in_resvcode => o_payload.resv_code,
in_resvHistorySequence => o_payload.resv_history_sequence );
COMMIT;
END queue_callback;
- I noticed that messages are not taken out of the 2nd queue,
I guess I would need to use the REMOVE option to delete messages from the queue?
Would this be a large source of performance degradation after just a few thousand messages?
- The data load through the DB may be a little bit intensive but I feel that doing things in parallel would help.
I would like to understand if Oracle has a way of dequeuing in parallel (with or without the use of notification)
In the case of multiple subscribers with notification , does 'job_queue_processes' value has an impact on the degree of parallelism? If not what setting has?
And is there a way supplied by Oracle to set the queue to notify only one subscriber per message?
Your advice would be very much appreciated
Philippe
Edited by: user528100 on Feb 23, 2009 8:14 AM