Counting rows in JDBC

Question

I've recently come across the problem of needing to count the number of rows in a potentially large resultset.
I did quite a bit of digging and found that the only suggested answers appear to be:
1. Do a SELECT COUNT(*) before doing the query.
2. Make the ResultSet scrollable, and use rset.last() and rset.getRow().
I have problems with both approaches:
1a. Two queries are issued rather than one - this could be quite expensive.
1b. The queries I am executing are dynamic - thus some clever logic is required to parse the statement and work out what we should be counting.
1c. The number of hits may change between the counting and the actual query.
2a. Definitely most simple, but the Oracle JDBC driver iterates through all the rows to get to the last one, meaning that the whole ResultSet is loaded in to the client-side cache, and thus killing the middle-tier unnecessarily.
2b. The results have to be scrollable - not a problem for me, but is too expensive for some.
I had settled on number 2 for a while, but performance was just too nasty when the ResultSet had more than a few million rows in it, and thus I've spent the last few days looking around for better alternatives.
select inner_2.*, rownum rowcount from (
select inner_1.*, (rownum*-1) neg_row, rownum pos_row from (
select COL1, COL2, COL3
from TABLE1, TABLE2
where TABLE1.COL1 = TABLE2.COL2
order by event_type
) inner_1
order by neg_row asc
) inner_2
order by pos_row asc
Let me explain. This query is determining the rowcount and returning the required columns all in one statement. All I need to do is look at the first row, and read the ROWCOUNT column (the last column) to know how many rows are in the resultset.
The disadvantage of this method is that an extra two sorts are required, and that there are three extra columns on the result set, which the caller didn't ask for. Hiding the extra three column is easy if you want a really clean solution.
These are definitely prices worth paying for me but I acknowledge that not everyone will find this price worth paying. However, having been immensely frustated with the lack of options for solving this problem, I thought I would share my solution with you all.
And on a less-altruistic note, if anyone can tell me how to optimise it further, has a better solution, or knows why it doesn't actually work like it appears to, I'd be more than interested to hear your comments.
Marcus

Answer 1

If you need a consistent resultset, you could create a global temp table-- each session would then query the temp table and see a consistent resultset over time. This has the disadvantage that the database has to duplicate the resultset for each session, which probably only moves the caching problem.
If your sessions don't go on forever and you have sufficient UNDO space, you can set the transaction isolation level to serializable. That will "freeze" the database to the point in time that the session begins. You need to have suffient UNDO, however, for the database to be able to reconstruct the database to that point in time and you need to be able to handle the situation where a user starts a session, walks away for the weekend, and comes back with an error indicating they need to reissue their query.
If you want to really get sophisticated, you can use Oracle Workspace Manager to "branch" the data for each user. Workspace Manager would allow you to keep a consistent view of data forever without duplicating data for each session.
The explain plan approach is useful as a rough estimate, not as an exact count. It's based on the statistics that the CBO is looking at, which ought to be close, but could easily be off by a factor of 2 or more depending on the complexity of the query. To get an actualt honest-to-goodness count, you need to incur the cost of a count(*). It seems to me, though, that if the data is particularly volitile, with the count constantly changing, it wouldn't be particularly useful to know the exact count at a point in time.
Note that the SQL chunking method I outlined above is much more efficient retrieving the first few pages than the last few. If the resultset is large, my assumption is that people aren't actually going page-by-page through the entire thing-- that would seem incedibly inefficient. If they're likely looking at the first few pages and the last few pages, you'd want to have slightly different queries for the first and last few pages-- the inner query's ORDER BY clause would have to be reversed (i.e. ASC for the first pages, DESC for the last few pages). Then, the middle pages would be the most expensive.
Justin
Distributed Database Consulting, Inc.
www.ddbcinc.com/askDDBC