Problem with Latches and Parallelising PL/SQL Cursors

Question

In 9i i have process that do the following:
PROCEDURE LoadClients IS
  CURSOR cChanges IS
    SELECT ...
    WHERE MOD(PK,pnTotalJobs>) = pnThisJob To call this I do the following
  nTOTALJOBS CONSTANT := 2;
BEGIN
  FOR i IN 1..nTOTALJOBS LOOP
    DBMS_JOB.SUBMIT(LoadClient(pnTotalJobs=>nTOTALJOBS, pnThisJob=>i-1));
  END LOOP;
  COMMIT;So by changing the value of the constant - nTOTALJOBS - I can control the number of instances of LoadClient that run. This means I only need one procedure which can be called multiple times, making maintenance simpler than having multiple procedures.
This process has been running successfully on several procedures of over a year. However recently I have been having trouble with latches in memory. I think these are related to using the variables in the cursor and part of the oracle treating the different queries as being identical. This results in locks in memory and none of the jobs running. In most cases I'm setting nTOTALJOBS to 2. I'm looking into the further with my DBAs.
My questions are:
1. Has anyone come across this sort of problem before?
2. If so is there something I can do in my code or DB parameters to alleviate or test this?
Notes.
1. The process uses row by row processing in PL/SQL This done to get row by row error handling. I'm not going to change this in the short-term
2. The SQL does a call over a database link, I don't know if this has any effect.
3. This method does have a proven performance benefit
4. The code snippets above are just illustrative, I'm not asking for syntax corrections.
Thank you for taking the time to read through this.

Answer 1

I would first investigate the "problem with latches" to be sure that it has anything to do with these jobs that you are submitting.
Your concern seems to be "because I am submitting multiple copies of the same PLSQL procedure to run concurrently, am I running into latching issues ?".
Take a step back and look at
a. MultiUser OLTP environments : At any time you can have more than a few users running the same SQL queries submitted through Forms / Client Frontends.
b. ERP Report Extractions (eg in Oracle EBusiness Suite or Peoplesoft) : At any time (say during the daily batch runs ?) you can have multiple copies of the same report (with different bind variables for different business_units / countries etc) running concurrently and extracting reports.
The question would be : How many concurrent executions of the same code is too much ?
I would say that latching issues come up on :
a. Shared Pool : Multiple sessions attempting to load SQL statements and Parse SQL statements
You could reduce some of this with reusable SQLs and bind variables instead of literals in quick running jobs / user front end screens etc
b. Buffer Cache : Hot Blocks being frequently pinned -- eg index root blocks or "busy" table blocks.
If you are running, say 4-8 concurrent copies of the the same code on a 4 CPU or 8 CPU machine, you may not be running into latching issues on the code -- "may not be" -- we can't be too sure, until we identify which latches are under "issue".
However, if the queries are executing nested loops and frequently accessing the same index blocks, you may more likely be having issues with latches on cache buffer chains.
So, this is what I would suggest :
a. Identify the size and scale of the "issues" -- how much CPU time / wait time is accounted for by the latches in relation to the total response time of each of the jobs
b. Identify which latches are the ones under contention
c. On the basis of which latches are the holding up performance, decide your next course of action.
Hemant K Chitale
http://hemantoracledba.blogspot.com