DB_GET_BOTH_RANGE fails when there is only one record for a key

Question

Using the DB_GET_BOTH_RANGE flag doesn't seem to work when there is a
single record for a key.
Here's a sample Python program that illustrates the problem. If you
don't know Python, then consider this to be pseudo code. :)
from bsddb3 import db
import os
env = db.DBEnv()
os.mkdir('t')
env.open('t',
db.DB_INIT_LOCK | db.DB_INIT_LOG | db.DB_INIT_MPOOL |
db.DB_INIT_TXN | db.DB_RECOVER | db.DB_THREAD |
db.DB_CREATE | db.DB_AUTO_COMMIT)
data = db.DB(env)
data.set_flags(db.DB_DUPSORT)
data.open('data', dbtype=db.DB_HASH,
flags=(db.DB_CREATE | db.DB_THREAD | db.DB_AUTO_COMMIT |
db.DB_MULTIVERSION),
txn = env.txn_begin()
#data.put('0', '6ob 0 rev 6', txn)
data.put('0', '7ob 0 rev 7', txn)
#data.put('0', '8ob 0 rev 8', txn)
data.put('1', '9ob 1 rev 9', txn)
txn.commit()
cursor = data.cursor()
print cursor.get('0', '7', flags=db.DB_GET_BOTH_RANGE)
cursor.close()
data.close()
env.close()
This prints None, indicating that the record who's key is '0' and
who's data begins with '7' couldn't be found. If I uncomment wither of
the commented out puts, so that there is more than one record for the
key, then the get with DB_GET_BOTH_RANGE works.
Is this expected behavior? or a bug?

Answer 1

You can use the DB_SET flag which will look for an exact key match. If
you use it with the DB_MULTIPLE_KEY flag, it will return multiple keys
after the point it gets a match. If you can post here how all your
LIKE QUERIES look, I may be able to provide you with the suited
combination of the flags you can use for them.I'm not doing like queries. I'm using BDB as a back end for an object
database. In the object database, I keep multiple versions of object
records tagged by timestamp. The most common query is to get the
current (most recent version) for an object, but sometimes I want the
version for a specific timestamp or the latest version before some
timestamp.
I'm leveraging duplicate keys to implement a nested mapping:
{oid -> {timestamp -> data}}
I'm using a hash access method for mapping object ids to a collection
of time stamps and data. The mapping of timestamps to data is
implemented as duplicate records for the oid key, where each record is
an 8-byte inverse timestamp concatenated with the data. The inverse
timestamps are constructed in such a way that they sort
lexicographically in inverse chronological order. So there are
basically 3 kinds of query:
A. Get the most recent data for an object id.
B. Get the data for an object id and timestamp.
C. Get the most recent data for an object id who's timestamp is before
a given timestamp.
For query A, I can use DB->get.
For query B, I want to do a prefix match on the values. This can be
thought of as a like query: "like <inverse-time-stamp>%", but it can
also be modelled as a range search: "get the smallest record that is >=
a given inverse time stamp". Of course, after such a range search,
I'd have to check whether the inverse time stamp matches.
For query C, I really want to do a range search on the inverse time
stamp prefixes. This cannot be modelled as a like query.
I could model this instead as {oid+timestamp -> data}, but then I'd
have to use the btree access method, and I don't expect that to scale
as I'll have hundreds of millions of objects.
We tried using BDB as a back end for our database (ZODB) several years
ago and it didn't scale well. I think there were 2 reasons for
this. First, we used the btree access method for all of our
databases. Second, we used too many tables. This time, I'm hoping that
the hash access method and a leaner design will provide better
scalability. We'll see. :)
If you want to start
on a key partial match you should use the DB_SET_RANGE flag instead of
the DB_SET flag.I don't want to do a key partial match.
Indeed, with DB_GET_BOTH_RANGE you can do partial
matches in the duplicate data set and this should be used only if you
look for duplicate data sets.I can't know ahead of time whether there will be duplicates for an
object. So, that means I have to potentially do the query 2 ways,
which is quite inconvenient.
As you saw, the flags you can use with cursor.get are described in
detailed here.But it wasn't at all clear from the documentation that
DB_GET_BOTH_RANGE wouldn't work unless there were duplicates. As I
mentioned earlier, I think if this was documented more clearly and
especially of there was an example of how one would work around the
behavior, someone would figure out that behavior wasn't very useful.
What you should know is that the usual piece of
information after which the flags are accessing the records, is the
key. What I advice you is to look over Secondary indexes and Foreign
key indexes, as you may need them in implementing your queries.I don't see how secondary indexes help in this situation.
BDB is
used as the storage engine underneath RDBMS. In fact, BDB was the
first "generic data storage library" implemented underneath MySQL. As
such, BDB has API calls and access methods that can support any RDBMS
query. However, since BDB is just a storage engine, your application
has to provide the code that accesses the data store with an
appropriate sequence of steps that will implement the behavior that
you want.Yup.
Sometimes you may find it unsatisfying, but it may be more
efficient than you think.Sure, I just think the notion that DB_GET_BOTH_RANGE should fail if
the number of records for a key is 1 is rather silly. It's hard for me
to imagine that it would be less efficient to handle the non duplicate
case. It is certainly less efficient to handle this at the application
level, as I'm likely to have to implement this with multiple database
queries. Hopefully, BDB's cache will mitigate this.
Thanks again for digging into this.
Jim