Advice with High Data Volume

Question

I have to make a process that gets like 100.000 records from a data base at onces and then i have to treat one by one. The method i use after executing a query puts all the data in a Recorset object.
I would like to know if the Recorset is going to be able to treat so many data fast or if thare is any other Class better to do this kind of programs. This process has to be as faster as possible.
I am alse wondering that maybe is faster to read several times from the data base in order to get less Data than reading only one tme an get all the data.
I am realy need to program this process the most optismt way posible.
If someone can help me or knows where can i find information about this topic i would realy apreciate it.
Thanks

Answer 1

You can do it faster if you avoid to transport data between database and the java program, do the process entirely inside de database.
define your java loop inside the database doing a complex sqlstatement or a stored procedure written in the database language... for example, if you are working with sql server then use a stored procedure with transact-sql.
A stored procedure allows you to call a program with parameters that resides into the database, and the database engine will execute it.
the statement to execute throught jdbc is something like:
"execute sp_my_stored_procedure param1, param2, ... , paramN"
where param1... paramN are all needed parameters to do your process, then inside the database must be a stored procedure it could be something like
create procedure sp_my_stored_procedure
@param1 type,
@param2 type
@param3 type
) AS
// HERE THE CODE of the procedure
declare my_cursor insensitive cursor for
SELECT column1, column2 ... columnN
where ...
open my_cursor
//get a row
fetch my_cursor into @column1, @column2 ... @columnN
while ( @@fetch_status = 0 ) begin
// process the row
// do operations on columns values with params
// i.e.
@result1 = @column1 - @column2 * @param1
// do operations with results of your ops ( update, insert... )
update anotherTable set value = @result1
where id = @column5
// get the next row
fetch my_cursor into @column1, @column2 ... @columnN
end
close my_cursor
allocate my_cursor
PD: use transactions if needed