Pagination support for non-Oracle databases?

Question

Hi,
I just read this thread (Pagination Support on pagination support. Is there any way to get pagination with non-Oracle databases? We are using an IBM iSeries / AS/400 DB2 database right now, and we're planning to use some local lightweight database in the near future as well (probably Cloudscape/Derby or "IBM Everyplace database".)
We currently use code like this:
String sql = "SELECT art FROM Artikel art" +
            /* dynamically generated where statement is added here */
            + "ORDER BY art.artikelNummer";
Query q = em.createQuery(sql);
q.setFirstResult(firstResult);
q.setMaxResults(maxResults);If I look in the TopLink logs, I see queries like this:
SELECT ARTNR, ARALT, ARAFJ, ARXII, ARAVJ, ARXIV, ARANJ, AHGCD, ARNVJ, ARCRJ, ARARK, ARFKJ, ARTNK, ARGP1, ASGCD, ARGP2, ARPR1, ARGP3, ARPR2, AREX1, ARPR3, AREX2, ARPR4, AREX3, ARASA, ARINA, ASSCD, ARIA1, ARBAN, ARIN1, ARBAV, ARIA2, ARBAK, ARIN2, ARCES, ARIA3, ARCDT, ARIN3, ARCRE, ARIA4, ARCWK, ARIN4, ARHBH, ARIA5, ARDFA, ARIN5, ARDFG, ARIA6, ARDOS, ARIN6, AREPW, ARINN, ARFOD, ARIAS, ARFOE, ARINS, ARFOF, ARNAB, ARFOI, ARNIB, ARFON, ARNIA, ARFOS, ARNN1, ARFTA, ARNA2, ARVIV, ARNO2, ARGAP, ARNN3, ARGPT, ARNA4, ARGPD, ARNO4, ARGPA, ARNN5, ARGPO, ARNA6, ARHIS, ARNN6, ARISP, ARNIO, ARKHM, ARNNS, MAGCD, AROVJ, MTGCD, ARPL1, ARMXM, ARPL2, MRKCD, ARPL3, ARMVR, ARVKJ, ARMIM, ARV12, ARMDT, ARVVJ, ARMTE, AR#VR, ARMTU, ARZLS, ARMTM, ARIAT, ARMWK, ARAVS, MPCCD, ARNVS, ARBTW, ARFJS, ARXI2, ARG2S, ARXI3, ARE1S, ARXI4, ARE3S, ARXI6, ARIB1, ARXI1, ARIB2, ARXI5, ARIB3, AROPI, ARIB4, ARPRV, ARIB5, SZGCD, ARIB6, ARSPC, ARINO, ARSMF, ARIOS, VEAAN, ARNIS, ARSYN, ARNO1, ARVR1, ARNA3, ARV1S, ARNN4, ARVR2, ARNO5, ARV2S, ARNIN, ARVR3, ARNOS, ARV3S, ARP1S, ARTFA, ARP3S, ARTFG, ARS12, ARUVC, ARZLD, ARUCW, ARAJS, ARBKV, ARCJS, ARVVI, ARG3S, ARVVP, ARINB, VPOCD, ARIO2, VPECD, ARIO4, ARVIH, ARIO6, ARVHG, ARNBS, ARVRW, ARNN2, ARVPR, ARNA5, ARVVR, ARNAS, ARVVS, ARP2S, ARVV1, ARSVV, ARZK1, ARNJS, ARNA1, ARNO3, ARIO1, ARNO6, ARIO5, AROJS, ARE2S, ARVJS, ARIBS, ARIAD, ARIO3, ARG1S FROM ART WHERE ((((ARUVC = 'N') AND (ARHIS = 'N')) AND (ASGCD = 7)) AND (AHGCD = 15)) ORDER BY ARTNR ASC
(Yeah, I know we have too much columns in the table...)
So, no pagination in the query. As you can see, we have a mechanism in place to dynamically generate a where clause. This is because the user can set filters. The problem is, if our user sets a filter that causes the result set to be significantly smaller, the performance is way better than when he sets no filter at all. We suppose this is because the whole result set is sent to TopLink, regardless of the values of firstResult and maxResults.
We are using TopLink Essentials 2.1-10, by the way.
Message was edited by:
Bart Kummel

Answer 1

Hi all,
I'm trying to subclass <tt>DatabasePlatform</tt> to add pagination support for the AS/400 DB2 database of my customer. To be fair, it is not going very well so far.
The first problem is, the query Chris found by googling (Re: Pagination support for non-Oracle databases?), does not work for AS/400s version of DB2. In fact, although it is called "DB2", the database on the AS/400 system is a whole other database than the "normal" DB2 database that runs on Windows and *nix. The AS/400 DB2 simply does not have a "ROW_NEXT" function.
Another option would be to use the <b>row_number() over()</b> mehtod. But, as can be read here, this function is only available from version V5R4 of OS/400. And guess what? We're stuck on V5R3 at this client. (We cannot upgrade, because there's an application in use that's written in Delphi and IBM dropped the Delphi binding from V5R4...)
So I pretty much ran out of options. On the mailing list I linked to above, someone mentions the option to make a sort of stored procedure that generates a row count number. An example of how to do this can be found here. I implemented it, and ended up with this code:
package com.myclientsname.persistence;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import org.eclipse.persistence.expressions.ExpressionBuilder;
import org.eclipse.persistence.internal.databaseaccess.DatabaseCall;
import org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter;
import org.eclipse.persistence.internal.expressions.SQLSelectStatement;
import org.eclipse.persistence.internal.sessions.AbstractSession;
import org.eclipse.persistence.logging.SessionLog;
import org.eclipse.persistence.platform.database.DatabasePlatform;
import org.eclipse.persistence.sessions.SessionProfiler;
public class AS400Platform extends DatabasePlatform {
    private static final long serialVersionUID = 0L;
    public AS400Platform(){
         super();
         super.setShouldBindAllParameters(false);
    public void printSQLSelectStatement(DatabaseCall call, ExpressionSQLPrinter printer, SQLSelectStatement statement) {
        int max = 0;
        int firstRow = 0;
        if (statement.getQuery()!=null){
            max = statement.getQuery().getMaxRows();
            firstRow = statement.getQuery().getFirstResult();
        if ( !(max>0) && !(firstRow>0) ){
            super.printSQLSelectStatement(call, printer,statement);
            return;
        } else {
            statement.setUseUniqueFieldAliases(true);
            ExpressionBuilder builder = new ExpressionBuilder();
            statement.addField(builder.getField("COUNTER() AS CNTR"));
            printer.printString("SELECT * FROM (");
            call.setFields(statement.printSQL(printer));
            printer.printString(") AS R WHERE R.CNTR >= ");
            printer.printParameter(DatabaseCall.FIRSTRESULT_FIELD);
            if ( max > 0 ){
                // Use of binding parameters is not allowed here, so use
                // String concatenation instead...
                printer.printString(" FETCH FIRST " + max + " ROWS ONLY");
        call.setIgnoreFirstRowMaxResultsSettings(true);
    public boolean wasFailureCommunicationBased(SQLException exception, Connection connection, AbstractSession sessionForProfile){
        if (connection == null || this.pingSQL == null){
            //Without a connection we are  unable to determine what caused the error so return false.
            //The only case where connection will be null should be External Connection Pooling so
            //returning false is ok as there is no connection management requirement
                //If there is no ping sql then we can not perform the ping.
            return false;
        PreparedStatement statement = null;
        try{
            sessionForProfile.startOperationProfile(SessionProfiler.ConnectionPing);
            if (sessionForProfile.shouldLog(SessionLog.FINE, SessionLog.SQL)) {// Avoid printing if no logging required.
                 sessionForProfile.log(SessionLog.FINE, SessionLog.SQL, getPingSQL(), (Object[])null, null, false);
            statement = connection.prepareStatement(getPingSQL());
            ResultSet result = statement.executeQuery();
            result.close();
            statement.close();
        }catch (SQLException ex){
            try{
                // Had to add this check because of NullPointerExceptions
                // (maybe a bug?)
                if(statement != null){
                    //try to close statement again in case the query or result.close() caused an exception.
                    statement.close();
            } catch (SQLException exception2) {
                //ignore;
            return true;
        }finally{
            sessionForProfile.endOperationProfile(SessionProfiler.ConnectionPing);
        return false;
}(As you can see, I had to override the <tt>wasFailureCommunicationBased()</tt> method as well, due to some unexpected NPE's. (A bug, perhaps?))
This code does work. However, the performance is not very well. The first page comes relatively fast, but as you browse further in the table, each page comes slower. I assume this is because the counter() method has to be evaluated for each row in the table.
I have to get the performance better and constant. Does anyone have an idea how to optimize this further?
Best regards,
Bart Kummel