Processing in chunks

Question

Hi,
I have a table with 3 million rows and I want to extract them into 100000 row chunks. there is a PK ID on the table and I can get the min and max PK ID and also use MOD to give me every 100000th PK ID like this:
select reference from (
select reference, rownum as rownumber from (
select reference from prop_cand_gen order by reference
) where MOD(rownumber-1,100000) = 0
order by reference
So now I need to come up with a query to extract the data based on the PKID ranges so something along the lines of:
select data1, data2 from table where refrence between 1 and 100000
select data1, data2 from table where reference between 100001 and 200000
right up to max(reference)
So I could right the 30 queries manually inputing the results from the mod query but i am after a more dynamic way to put this through a loop[ and generate the queries from MIn to MAx in chunks of 100000.
Any ideas welcome :0

Answer 1

Hi,
Welcome to the forum!
Think carefully about why you need to divide this into chunks.
If you have a good reason, you can write a query that will generate the 30 (or however many) SQL statements for you. For example:
WITH     got_r_num          AS
     SELECT  rfrnc          -- REFERENCE isn't a very good column name
     ,     ROW_NUMBER () OVER (ORDER BY rfrnc)     AS r_num
     FROM     pop_cand_gen_order
SELECT       'SELECT data1, data2 FROM  table_x WHERE rfrnc >= '
       || rfrnc
       || CASE
                  WHEN  LEAD (rfrnc) OVER (ORDER BY rernc) IS NOT NULL
           THEN  ' AND rfrnc ~ ' || LEAD (rfrnc) OVER (ORDER BY rernc) - 1
--              This character  ^  is supposed to be a < sign
              END
       || ';'
FROM      got_r_num
WHERE       MOD (r_num, 100000) = 1
ORDER BY  r_num
;Edited by: Frank Kulash on Oct 2, 2012 11:06 AM
Used &gt;= and &lt; instead of BETWEEN
Edited by: Frank Kulash on Oct 2, 2012 11:11 AM
This site is getting confused by the < sign in a string literal.
Use <, not ~, to try this