Alternatives to using update to flag rows of data

Question

Hello
I'm looking into finding ways of improving the speed of a process, primarily by avoiding the use of an update statement to flag a row in a table.
The problem with flagging the row is that the table has millions of rows in it.
The logic is a bit like this,
Set flag_field to 'F' for all rows in table. (e.g 104million rows updated)
for a set of rules loop around this
Get all rows that satisfy criteria for a processing rule. (200000 rows found)
Process the rows collected data (200000 rows processed)
Set flag_field to 'T' for those rows processed (200000 rows updated)
end loop
Once a row in the table has been processed it shouldn't be collected as part of any criteria of another rule further down the list, hence it needs to be flagged so that it doesn't get picked up again.
With there being millions of rows in the table and sometimes rules only processing 200k rows, i've learnt recently that this will create a lot of undo to be written and will thus take a long amount of time. (any thoughts on that)
Can anyone suggest anything other then using an update statement to flag each row to avoid it being processed again?
Appreciate the help

Answer 1

With regard to speeding up the process, the answer is
"it depends". It all hinges on exactly what you mean
by "Process the rows collected data". What does the
processing involve?the processing involved is very straight forwards.
data in these large tables stay unchanged. for sake of this example, i'll call this table orig_data_table, the rules are set by users who have their own tables built (for this example we'll call it target_data_table).
the rules are there to take rows from the orig_data_table and insert them into the target_data_table.
e.g
update orig_data_table set processed='F';rule #1 states : select * from orig_data_table where feature_1 = 310;this applies to 3000 rows in orig_data_table.
these 3000 rows get inserted to target_data_table.
final step is to flag these 3000 rows in the orig_data_table so that they are not used by another rule proceeding rule 1 : update orig_data_table.processed='T' where feature_1 = 310;
commit;rule #2 states :select * from orig_data_table where feature_1=310 and destination='Asia' and orig_data_table.processed = 'F'so it won't pick the 3000 rows that were processed as part of rule #1
once rule #2 has got the rows from orig_data_table (e.g 400000 rows)
those get inserted to the target_data_table, followed by them being flagged to avoid being retrieved again
update orig_data_table.processed='T' where destination='Asia';
commit;continue onto rule #3...
- Is the process some kind of transformation of the
~200,000 selected rows which could possibly be
achieved in SQL alone, or is it an extremely complex
transformation which can not be done in pure SQL?its not at all complex, as i say the data in orig_data_table is unchanged bar the processed field which is initially set to 'F' for all rows and then set to 'T' after each rule for those rows fulfilling the criteria of each rule.
- Does the FLAG_FIELD exist purely for the use of
this process or is it referred to by other
procedures?the flag_field is only for this purpose and not used elsewhere
Having said that, as a first step to simply avoid the
update of the flag field, I would suggest that you
use bulk processing and include a table of booleans
to act as the indicator for whether a particular row
has been processed or not.could you elaborate a bit more on this table of booleans for me, it sounds an interesting approach for me to test...
many thanks again
Sandip