Associative array comparison and INSERT upon IF condition

Question

Hi Guys,
I have written this pl sql code to identify non existing sellers and insert their sales channel information into the dimension table (dimensional table update).
Somehow,......nothing is inserted and this script runs for 12 hours+ without any result. the sql autotrace shows no result and the explain plan (button on sql developer throws upon clicking "missing keyword". I have no
information what is going on/wrong. Does anyone spot an error?
UNDEFINE DimSales;
UNDEFINE FactTable;
DEFINE DimSales = 'testsales';
DEFINE FactTable = 'testfact';
DECLARE
v_SellerNo VarChar(9);
v_error_code T_ERRORS.v_error_code%TYPE;
v_error_message T_ERRORS.v_error_message%TYPE;
TYPE assoc_array_str_type1 IS TABLE OF VARCHAR2(32) INDEX BY PLS_INTEGER;
     v1 assoc_array_str_type1;
TYPE assoc_array_str_type2 IS TABLE OF VARCHAR2(32) INDEX BY PLS_INTEGER;
     v2 assoc_array_str_type2;
BEGIN
--Collect all distinct SellerNo into associative array (hash table)
select distinct SellerNo bulk collect into v1 from &FactTable;
select distinct seller_id bulk collect into v2 from &DimSales;
v_SellerNo := v1.first;
loop
exit when v1 is null;
--1 Check if v_SellerNo already exists in DIM_Sales (if NOT/FALSE, its a new seller and we can insert all records for that seller
if (v2.exists(v_SellerNo)=false) THEN
INSERT INTO &DimSales (K_Sales,REG,BVL,DS, VS,RS,GS,VK)
(SELECT DISTINCT trim(leading '0' from RS||GS) ,REG BVL,DS,VS,RS,GS,VK from &FactTable where SellerNo =v_SellerNo);
--ELSE
end if;
v_SellerNo := v1.next(v_SellerNo);
end loop;
EXCEPTION
WHEN OTHERS THEN
ROLLBACK;
--v_error_code := SQLCODE
--v_error_message := SQLERRM
--INSERT INTO t_errors VALUES ( v_error_code, v_error_message);
END;
---------------------------------------------------------------

Answer 1

Distinct clause requires a sort. Sorts can be very expensive.
Bulk collects that are not constrained in fetch size, can potentially fetch millions of rows - requiring that data to be wholly read into server memory. I have seen how this can degrade performance so badly that the kernel reboots the server.
Using PL/SQL loops to process and insert/update/delete data is often problematic due to its row-by-row approach - also called slow-by-slow approach. It is far more scalable letting SQL do the "loop" processing, by using joins, sub-selects and so on.
Where the conditional processing is too complex for SQL to handle, then PL/SQL is obviously an alternative to use. Ideally one should process data sets as oppose to rows in PL//SQL. Reduce context switching by using bulk fetches and bulk binds.
But PL/SQL cannot execute in parallel as the SQL it fires off can. If after all the optimisation, the PL/SQL process still needs to hit a million rows to process, it will be slow irrespective of how optimal that PL/SQL approach and design - simply because of the number of rows and the processing overheads per row.
In that case, the PL/SQL code itself need to be parallelised. There are a number of ways to approach this problem - the typical one is to create unique and distinct ranges of rows to process, spawn multiple P/SQL processes, and provide each with a unique range of rows to process. In parallel.
So you need to look close at what you are trying to achieve, what the workloads are, and how to effectively decrease the workloads and increase the processing time of a workload.
For example - finding distinct column values. You can pay for that workload when wanting that distinct list. And each time afterward repeat that workload when wanting that distinct list. Or you can pay for that workload up-front with the DML that creates/updates those values - and use (for example) a materialised view to maintain a ready to use distinct list of values.
Same workload in essence - but paying once for it and up-front as oppose to each time you execute your code that needs to dynamically build that distinct list.
Kent Crotty did tests and showed stunning performance improvements with bulk collect and forall, up to 30x faster:Bulk processing is not a magical silver bullet. It is a tool. And when correctly use, the tool does exactly what it was designed to do.
The problem is using a hammer to drive in screws - instead of a screwdriver. There's nothing "stunning" about using a screwdriver. It is all about using the correct tool.
If the goal of the swap daemon is to free up "idle" chunks of memory, and try to use that memory for things like file cache instead, what does that have to do with bulk processing?The swap daemon reads virtual memory pages from swap space into memory, and writes virtual pages from memory to swap space.
What does it have to do with bulk processing? A bulk fetch reads data from the SGA (buffer cache) into the PGA (private process memory space). The larget the fetch, the more memory is required. If for example 50% of server memory is required for a bulk collection that is 2GB in size, then that will force in-use pages from memory to swap space.. only to be swapped back again as it is needed, thereby forcing other in-use pages to swap. The swap daemon will consume almost all the CPU time swapping hot pages continually in and out of memory.