How to get primary keys in some order with joins?

Question

Hi, I build BBS using BDB as backend database, forum database, topic database and post database share one environment. This BBS web application is multi-thread program. If user selects one forum, its topics will be listed in order of last reply time;selecting one topic, posts are listed in order of reply time as well.
struct forum {
UInt16 forumID;
string forumName;
string _lastPoster;      // who is the last one replied in this forum
struct topic {
UInt32 topicID;
UInt16 forumID; // topic comes from this forum
string title; // topic title
UInt64 dateOfLastReply; // when last reply to this topic happen
struct post {
UInt64 postID;
UInt32 topicID; // post comes from this topic
string title; // post title as of topic
UInt64 dateOfPost; // when this post is created
I create one primary database and two secondary databases for topic, primary key is topicID, secondary key are forumID and dateOfLastReply respectively, and I want to show 1st 25 topics in latest reply time order on the 1st browser page, 2nd 25 topics on the 2nd browser page, and etc.
if using SQL, it will be: SELECT topicID FROM topic WHERE forumID=xx ORDER BY dateOfLastReply DESC
From performance perspective, I want get all topics id of one same forum, and need them come in reply time order, then retrieve topic one by one based on returned topicID, how can I do this? guess I have to use joins.
Plus, do you have any suggestion about retrieval performance given the fact that topics retrieval will happen each time browser want to request the next page, that is, 2nd 25 topics of this forum?
Is DB_DBT_MULTIPLE helpful to me?
thanks.
Edited by: tiplip on 2011-1-22 上午5:43
Edited by: tiplip on 2011-1-22 下午5:52
Edited by: tiplip on 2011-1-23 下午7:42

Answer 1

Hi tiplip,
Bellow I will describe how you can support "SELECT * FROM table WHERE X = key ORDER BY Y" queries using Berkeley DB, which, as you suspected, should be done by using a composite index.
First of all, think of Berkeley DB as the storage engine underneath an RDBMS. In fact, Berkeley DB was the first "generic data storage library" implemented underneath MySQL. As such, Berkeley DB has API calls and access methods that can support any RDBMS query. However, since BDB is just a storage engine, your application has to provide the code that accesses the data store with an appropriate sequence of steps that will implement the behavior that you want.
If you have two indices in SQL, each on a single column (call them X and Y), and you do:
SELECT * FROM table WHERE X = key ORDER BY Y;then there are three plausible query plans:
(1) scan the whole table, ignore both indices, filter by X = key then sort by Y;
(2) use the index on Y to scan all rows in the required order, filter by X = key;
(3) use the index on X, find the matching rows, then sort by Y.
There are cases where (1) would be fastest, because it has all of the columns from one scan (the other query plans will do random lookups on the primary for each row). This assumes that the data can fit into memory and the sort is fast.
Query plan (2) will be fastest if the selectivity is moderate to high, looking up rows in the main table is fast, and sorting the rows is very slow for some reason (e.g., some complex collation).
Query plan (3) will be fastest if the selectivity is small (only a small percentage of the rows in the table matches). This should be the best case for us, making it the best choice in a Berkeley DB key/value application.
The optimal plan would result from having a composite index on (X, Y), which can return just the desired rows in the desired order. Of course, it does cost additional time and space to maintain that index. But note that you could have this index instead of a simple index on X: it can be used in any query the simple index could be used in.
Records in Berkeley DB are (key, value) pairs. Berkeley DB supports only a few logical operations on records. They are:
* Insert a record in a table.
* Delete a record from a table.
* Find a record in a table by looking up its key.
* Update a record that has already been found.
Notice that Berkeley DB never operates on the value part of a record. Values are simply payload, to be stored with keys and reliably delivered back to the application on demand. Both keys and values can be arbitrary byte strings, either fixed-length or variable-length.
So, in case of a "SELECT * FROM X WHERE id=Y ORDER BY Z" query, our suggestion, from Berkeley DB's point of view, would be for you to use a composite index (as it would be in SQL), where a string created as X_Y should do the trick, as explained in the following scenario.
Primary:
    X Y
1 10 abc
2 10 aab
3 20 bbc
4 10 bba
5 20 bac
6 30 cbaSecondary:
10_aab 2
10_abc 1
10_bba 4
20_bac 5
20_bbc 3
30_cba 6If the query looks like this:
'SELECT * FROM primarydb WHERE X = 10 ORDER by Y'the application can run a cursor on the secondary and begin the loop with the DB_SET_RANGE flag on 10. When iterating with DB_NEXT, this will return:
2 10 aab
1 10 abc
4 10 bbcThe application must check for the end of the range inside the loop, in this case it should stop when it hits 20_bac.
As in SQL, retrieving by a secondary key is remarkably similar to retrieving by a primary key and the Berkeley DB call will look similar to its primary equivalent.
tiplip wrote:
Plus, do you have any suggestion about retrieval performance given the fact that topics retrieval will happen each time browser want to request the next page, that is, 2nd 25 topics of this forum?As you are concerned about the performance, I think this would be the fastest solution. Of course, you can tune the performance at a later time, after you have the functionality in place. What I think you should do first is to increase the cache size and test with a bigger database page size, and maybe to configure the transactional subsystem (in case you use one).
If you are not very familiar with how to implement the above in BDB, please read the Guide to Oracle Berkeley DB for SQL Developers, available at: http://www.oracle.com/technetwork/articles/seltzer-berkeleydb-sql-086752.html
You will also need to be familiar with the following documentation:
Related documentation pages:
Secondary indexes - http://download.oracle.com/docs/cd/E17076_01/html/programmer_reference/am_second.html
Cursor operations - http://download.oracle.com/docs/cd/E17076_01/html/programmer_reference/am_cursor.html#am_curget
DBcursor->get() - http://download.oracle.com/docs/cd/E17076_01/html/api_reference/C/dbcget.html
DB_SET_RANGE - http://download.oracle.com/docs/cd/E17076_01/html/api_reference/C/dbcget.html#dbcget_DB_SET_RANGE
If my answer helps you with your question, please go ahead and rate it as Helpful or Correct, and the forum thread as answered. For each unrelated question, please create a new forum thread.
Good luck with building your forum application,
Bogdan Coman
PS: If you are a BDB licensed customer, you can also use My Oracle Support (https://support.oracle.com) to visit the KM note 1210173.1, that discusses the same topic.