Displaying large result sets in Table View u0096 request for patterns

Question

When providing a table of results from a large data set from SAP, care needs to be taken in order to not tax the R/3 database or the R/3 and WAS application servers.  Additionally, in terms of performance, results need to be displayed quickly in order to provide sub-second response times to users.
This post is my thoughts on how to do this based on my findings that the Table UI element cannot send an event to retrieve more data when paging down through data in the table (hopefully a future feature of the Table UI Element).
Approach:
For data retrieval, we need to have an RFC with search parameters that retrieves a maximum number of records (say 200) and a flag whether 200 results were returned. 
In terms of display, we use a table UI Element, and bind the result set to the table.
For sorting, when they sort by a column, if we have less than the maximum search results, we sort the result set we already have (no need to go to SAP), but otherwise the RFC also needs to have sort information as parameters so that sorting can take place during the database retrieval.  We sort it during the SQL select so that we stop as soon as we hit 200 records.
For filtering, again, if less than 200 results, we just filter the results internally, otherwise, we need to go to SAP, and the RFC needs to have this parameterized also.
If the requirement is that the user must look at more than 200 results, we need to have a button on the screen to fetch the next 200 results.  This implies that the RFC will also need to have a start point to return results from.  Similarly, a previous 200 results button would need to be enabled once they move beyond the initial result set.
Limitations of this are:
1.     We need to use custom RFC function as BAPI’s don’t generally provide this type of sorting and limiting of data.
2.     Functions need to directly access tables in order to do sorting at the database level (to reduce memory consumption).
3.     It’s not a great interface to add buttons to “Get next/previous set of 200”.
4.     Obviously, based on where you are getting the data from, it may be better to load the data completely into an internal table in SAP, and do sorting and filtering on this, rather than use the database to do it.
Does anyone have a proven pattern for doing this or any improvements to the above design?  I’m sure SAP-CRM must have to do this, or did they just go with a BSP view when searching for customers?
Note – I noticed there is a pattern for search results in some documentation, but it does not exist in the sneak preview edition of developer studio.  Has anyone had in exposure to this?
Update - I'm currently investigating whether we can create a new value node and use a supply function to fill the data.  It may be that when we bind this to the table UI element, that it will call this incrementally as it requires more data and hence could be a better solution.

Answer 1

Hi Matt,
i'm afraid, the supplyFunction will not help you to get out of this, because it's only called, if the node is invalid or gets invalidated again. The number of elements a node contains defines the number of elements the table uses for the determination of the overall number of table rows. Something quite similar to what you want does already exist in the WD runtime for internal usage. As you've surely noticed, only "visibleRowCount" elements are initially transferred to the client. If you scroll down one or multiple lines, the following rows are internally transferred on demand. But this doesn't help you really, since:
1. You don't get this event at all and
2. Even if you would get the event, since the number of node elements determines the table's overall rows number, the event would never request to load elements with an index greater than number of node elements - 1.
You can mimic the desired behaviour by hiding the table footer and creating your own buttons for pagination and scrolling.
Assume you have 10 displayed rows and 200 overall rows, What you need to be able to implement the desired behaviour is:
1. A context attribute "maxNumberOfExpectedRows" type int, which you would set to 200.
2. A context attribute "visibleRowCount" type int, which you would set to 10 and bind to table's visibleRowCount property.
3. A context attribute "firstVisibleRow" type int, which you would set to 0 and bind to table's firstVisibleRow property.
4. The actions PageUp, PageDown, RowUp, RowDown, FirstRow and LastRow, which are used for scrolling and the corresponding buttons.
The action handlers do the following:
PageUp: firstVisibleRow -= visibleRowCount (must be >=0 of course)
PageDown: firstVisibleRow += visibleRowCount (first + visible must be < maxNumberOfExpectedRows)
RowDown/Up: firstVisibleRow++/-- with the same restrictions as in page "mode"
FirstRow/LastRow is easy, isn't it?
Since you know, which sections of elements has already been "loaded" into the dataSource-node, you can fill the necessary sections on demand, when the corresponding action is triggered.
For example, if you initially display elements 0..9 and goto last row, you load from maxNumberOfExpected (200) - visibleRows (10) entries, so you would request entries 190 to 199 from the backend.
A drawback is, that the BAPIs/RFCs still have to be capable to process such "section selecting".
Best regards,
Stefan
PS: And this is meant as a workaround and does not really replace your pattern request.