Determining size of CursoredStream

Question

Hello,
we are developing a system using Toplink. One of our functions uses a CursoredStream to read the result set in sections from the server. The data read query is built "manually" due to complexity reasons. The query needs an additional count query to determine the size of the result set. Now we found that the execution of this count query causes a loss of performance. Is there any other way to determine the result set's size avoiding the count query to be executed? We tried to loop through the CursoredStream to count its elements but this is even slower for large result sets...
Any help is appreciated!
Thank you in advance!
Danilo Gld_l

Answer 1

Hello,
thank you for your answer. By now, I tried the following:
(1) Loop through the CursoredStream and count its objects:
     aCursoredStream = (CursoredStream) getClientSession().executeQuery(aReadAllQuery);
     int size = 0;
     aCursoredStream.mark(1);
     while (!aCursoredStream.atEnd()) {
          aCursoredStream.read();
          size++;
     aCursoredStream.setSize(size);
     aCursoredStream.reset();
(2) Loop through the CursoredStream and read its object in sections:
     aCursoredStream = (CursoredStream) getClientSession().executeQuery(aReadAllQuery);
     int size = 0;
     int stepSize = 100; // also tried other values without mentionable performance differences
     aCursoredStream.mark(1);
     while (!aCursoredStream.atEnd()) {
          try {
               aCursoredStream.read(stepSize);
          } catch (QueryException exc) {
               stepSize = stepSize / 2;
               continue;
          size += stepSize;
     aCursoredStream.setSize(size);
     aCursoredStream.reset();
(3) Filter CursoredStream size from Exception message:
     aCursoredStream = (CursoredStream) getClientSession().executeQuery(aReadAllQuery);
     int size = 0;
     int stepSize = Integer.MAX_VALUE - 1;
     // initialize field for "position" attribute of the cursored stream
     Field positionField = null;
     // store page size of cursored stream's policy
     int pageSize = aCursoredStream.getPolicy().getPageSize();
     aCursoredStream.mark(1);
     // set policy's page size to maximal integer to avoid exception when changing position and reading
     aCursoredStream.getPolicy().setPageSize(stepSize + 1);
     try {
          // fetch "position" attribute of the cursored stream and make it accessible
          positionField = aCursoredStream.getClass().getDeclaredField("position");
          positionField.setAccessible(true);
     } catch (NoSuchFieldException exc) {
          exc.printStackTrace();
     // while cursor is not at the stream's end yet (should never happen)     
     while (!aCursoredStream.atEnd()) {
          try {
               // set "position" to actual position (should always be 0) plus step
               // must happen via reflection because there is not any setter available
               positionField.set(aCursoredStream,
                              new Integer(aCursoredStream.getPosition() + stepSize));
               try {
                    // try to read the next element from the stream
                    aCursoredStream.read();
               } catch (QueryException exc) {
                    // exception since stream size is not greater than the initial read size of
                    // cursor policy
                    // reset the cursor and the size counter
                    aCursoredStream.reset();
                    size = 0;
                    // while cursor is not at the stream's end yet
                    while (!aCursoredStream.atEnd()) {
                         // read an element from the stream
                         aCursoredStream.read();
                         // and increment the size counter
                         size++;
                    // leave the outer while loop
                    break;
          } catch (IllegalAccessException exc) {
               exc.printStackTrace();
          } catch (ArrayIndexOutOfBoundsException exc) {
               // exception since element at position of maximal integer does not exist
               // filter size from exception message and leave while loop
               size = new Integer(exc.getMessage().
                              substring(exc.getMessage().lastIndexOf(" ") + 1)).intValue();
               break;
     // if size could be determined, provide it to cursored stream
     if (size != 0) {
          aCursoredStream.setSize(size);
     // reset cursor policy's page size to initial value
     aCursoredStream.getPolicy().setPageSize(pageSize);
     // reset cursor position
     aCursoredStream.reset();
The third option proved to be a little bit faster than the other two. But it is still slower than the COUNT(*) query for large result sets depending on the query. We are talking about result sets comprising some thousand records. Depending on the complexity of the query, the count query takes longer than the data query - for whatever reasons...
The data query always contains several ORDER BY criteria. So, in my opinion, the database has to process the complete result set (or at least all entries of the ORDER BY columns) to correctly sort it. Am I wrong if I assume that it has to know the record count afterwards? My intention is to retrieve this size with the least performance effort...
Nevertheless I will try your suggestion to retrieve the count by executing a ReportQuery. Thank you.
At the moment I am trying to use a ScrollableCursor to retrieve its size as follows:
     if (aScrollableCursor.last()) {
          aScrollableCursor.setSize(aScrollableCursor.currentIndex());
          aScrollableCursorResultSet.setSize(aScrollableCursor.getSize());
          aScrollableCursor.first();
Performance tests are pending...
Danilo Gld_l