What is the MAX amount of DATA for JDBC?

Question

Hello,
Did anybody had any expericence with how jdbc works with large amount of data SAY
> 1/2 GIG or more?
Any help apprecieated
Thanks.
Paul

Answer 1

Always depends on the driver implementation. Each
driver/database combination deals with massiveamounts
of data in a single transaction differently. Canyou
be more specific? In general Most Drivers of
commercial grade quality and/or shipped with the
top
5
RDBMS have pretty solid implementations limitedsoley
by network bandwidth to carry this info. A good
developer however will not rely on the driver to
manage this much info and will develop a scheme
to
minimize the impact of such large data transferson
an
unsuspecting database or client.
LThis is what I thought. I just need moreinformation
if for example I have a JAVA process which feedsdata
from RDMBS and the amount of data the processreceives
SAY 2GIG My guess this all get in "ResultSet"object
and thus all feeds in VM( memory) so I have to
designing my process SUCH that it takes only asmall
sample of data and iterate this until I get all the
information from my query. Ex.
select * from test where data between 1 and 2000000;I
have to brake this down by "DAYS"
so my code will be something like
for (int i=0 ;i<2000000 ;i++) {
String select=null;
select="select * from test where data="+i+;
do( something else)
Am I close far? Is there another way that may beJDBC
driver handless this behind scenes?
Well personally I think you should be structuring your
SQL better than that. Making 2000000 calls over a
network in a for loop is suicidal..and lets not forget
how something like that will block your application.
First you can do something like "select * from test
where data <= 2000000" in one shot. Now you get the
whole set and you can scroll through and do your
operation ( of course this is slow too, but its much
better than running the query 2000000 times ).
the other thing you have to consider is : is it
necessary to really block during an operation like
this? Can it be done in another thread? a daemon
process even? Some service which can be polled for
results when needed?
This is a tricky example and I'd be interested to know
more about what you are trying to do.
When you were talking about large data I was under the
impression of something like a blob, clob, or any sort
of single query with 2gb of data in the result.
Can you break up your query? Doing things in batch in
an alternative thread can yield to other processes
perhaps?
Loh, and also: Can the Database itself take care of this business? Stored Procedures and Internal Functions are all built to handle massive amounts of data with minimal impact. Can you leverage the technology within the Database to accomplish your goal? Sometimes a well written procedure can save you tons of work in your business logic tier.
The right tools for the right job..
L