Cursor memory use in a background job

Question

I am researching a solution for a client, and need help as I'm admittedly an Oracle greenhorn. They run 10g Enterprise Edition.
{color:navy}*The problem*{color}: create a low-priority background process that uses DB inactivity windows to read through a large table (over 25 million records) a row at a time, and perform some simple operations with a few fields from each row. The process needs to be running 24x7 by default, although able to be administratively paused/restarted at any time. It also needs to pause itself M hours after last row read, and restart after a subsequent N hours.
h3.
Oracle Jobs
First, I'm wondering whether this sounds like a task suitable for a "job" managed by the Scheduler (<tt>DBMS_SCHEDULER</tt>), which could be enabled or disabled to mimic pausing/restarting functionality.
h3.
Cursors and memory usage
My second, main question lies with memory usage. One of their main concerns is that the background process not occupy much memory on the server. The general flow I've worked up:
<li> Function1: declare and open a cursor for the main SELECT. This would be run once only. Call Function2.
<li> Function2: the second, main function would perform a fetch of one row on the cursor, and make a call to an auxiliary function to operate on the data. Function2 would be executed constantly as the main process.
<li> Function3: close the cursor. Would be executed upon reaching the last row of the table, or if for some reason the admin. should decide to abort the process prematurely.
<li> If the client decides such, this process may be constant, so it would be restarted automatically with a call back to Function1.
However, I don't understand cursors enough to know if that would occupy a lot of memory on the server throughout the life-cycle of the master process, knowing ahead of time that the resultset the cursor identifies is very large. I've attempted to research this both in manuals (the DB Administrator, SQL Reference, Application Developer's Guide) and these forums, but not able to track down a clear answer -- I got lost somewhere between PGA and SGA memory. The furthest I've gotten in understanding cursors is that they're a mere instruction set, not a copy of the resultset, and therefore don't occupy a substantial amount of resident memory. I would very much appreciate some verification about whether this is accurate or not.
I'm wide open to suggestions -- thanks in advance for the help!

Answer 1

Wow, so many good questions make me only think of more questions! Part of the problem is I don't get to talk directly to the client -- any questions I have (such as statistical data about the main_table) must be formed to my manager, who relays them to the client and then we wait for an answer.
Toine wrote:
Are these 'DB inactivity windows' predictable time windows or do you mean that the process should only eat up resources when activity on the database server is low?Not predictable timeframes, but rather the process should take advantage of database server inactivity.
acadet wrote:
For the large table, how does it get its data and what happens to a row once it has been processed? Is the data static or constantly changing and if it does change, does it happen in a batch or in a series of small update/inserts etc.To my best knowledge, and without compromising the client's security: _the main table represent debts_. This is a financial institution, and it is their primary table. Each record contains about 100 fields which include various dates, amounts, rates, etc. Unfortunately, as I commented before, this table does not have a numeric primary key, but rather the 2 fields (one numeric and one varchar2) which serve together as unique identifiers. It's not a static table, as new debts are added a bit at a time for new clients, etc. Also, the existing debts are updated when rates change, etc. I've requested statistical information on the extent to which the data is updated.
Billy Verreynne wrote:
All SQLs that are parsed and executed in Oracle, is done as cursors. Do not confuse that with how the client deals with that cursor, explicitly, implicitly, via a cursor or ref cursor handle, etc. SQL = cursor.
Andre raised an important point - it sounds like what should be business/functional specifications having technical detail included that needs to be part of the solution. Been in a few boardroom brawls myself over that - telling clients that they must state WHAT needs to be done, and not technically HOW it will be done. Worse offenders usually are senior or executive management that used to be techies decades ago and still think that they are somehow still technically competent at a software engineering level.Of course I was referring to an explicit cursor when I asked about disabling consistent read. I'm very doubtful that this is a good idea any more.
Another challenge I have with this client is that since they are a large company and actually have their own development team, the requirements come from different members of that team. Depending who wrote them, they are better or more poorly stated, and often mix in detailed technical requirements framing a boxed solution. You would expect the management to revise these requirements and weed out the nonessentials that we should be able to propose ourselves... go figure. <steps down from soapbox>
I know it's hard for everyone to understand the main requirements when I've piece-mealed the problem description together. I'm just concerned about not revealing too much information that might even possibly compromise the client.
How does the following solution sound:
I've discarded the "explicit big-momma cursor that does it all ;)" possibility, as it presents way too many issues. Instead, we can propose to the client that we add a column to the debts_table which tracks processing status, 0 or 1. 0 if it hasn't been processed yet or errors occur in processing, and 1 once it's processed successfully. 0 would be the default for all existing records and on addition of new debts. In this way, all I would have to do is select one row at a time, possibly in order of the debt's addition date, and if its processing status is 0 then process it. If it completes successfully, update the status to 1. Flow would return back to the select statement, which would select the next row with status=0, and so on.
Edited by: CrackerJackhammer on Aug 5, 2009 2:43 AM