Queue method does not recycle the extent file

Question

Hi, all,
I use Queue methond in TDS of db-4.7.25, and enqueue with set(DB_APPEND) and inqueue with get(DB_CONSUME).
I configured the queue db with set_q_extentsize.
I found there are still extent files in env dir(see __dbq.test1.20 and __dbq.test1.21), though the queue is empty.
-rw-r----- 1 root root 24K Dec 9 17:22 __db.001
-rw-r----- 1 root root 25M Dec 9 17:22 __db.002
-rw-r----- 1 root root 321M Dec 9 17:22 __db.003
-rw-r----- 1 root root 4.1M Dec 9 17:22 __db.004
-rw-r----- 1 root root 33M Dec 9 17:22 __db.005
-rw-r----- 1 root root 15M Dec 9 17:22 __db.006
-rw-r--r-- 1 root root 64M Dec 9 17:25 __dbq.test1.20
-rw-r--r-- 1 root root 64M Dec 9 17:26 __dbq.test1.21
-rw-r--r-- 1 root root 40M Dec 9 17:32 __dbq.test1.36
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000333
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000334
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000335
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000336
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000337
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000338
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000339
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000340
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000341
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000342
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000343
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000344
-rw-r----- 1 root root 10M Dec 9 17:28 log.0000000345
-rw-r----- 1 root root 10M Dec 9 17:38 log.0000000346
-rw-r--r-- 1 root root 4.0K Dec 9 17:29 test1
# /usr/local/BerkeleyDB.4.7/bin/db_stat -d test1
Wed Dec 9 17:41:47 2009     Local time
42253     Queue magic number
4     Queue version number
1024     Fixed-length record size
0x20     Fixed-length record pad
4096     Underlying database page size
16384     Underlying database extent size
0     Number of records in the database
1     Number of database pages
3072     Number of bytes free in database pages (25% ff)
1800002     First undeleted record
1800002     Next available record number
This issue confuses me, does it have a delay recycling of extent files? But why only 20 and 21? or Is this a bug of 4.7.25 version?
Regards,
Steve

Answer 1

Hi Steve,
Any chance you might have deleted records in the queue database by record number and not only through DB_CONSUME operations? Have you used the DB->del method or just DB->get with the DB_CONSUME flags?
In general, only if the data is deleted using a DB_CONSUME operation can BDB be sure of removing a queue extent file. Otherwise, if BDB detects that a record at the head of the queue has been deleted, then it will check whether the extent file can be deleted. Though, active concurrent updates in the queue might prevent BDB from detecting that.
In BDB 4.8.24 the DB_CONSUME flag is set on all deletes in the queue, hence forcing the check to see if the queue extent can be deleted.
However, this is not done all the time as it affects concurrency.
I believe you're seeing the effects of bug #17004. This is fixed in BDB 4.8.24, and the relevant entry in the change log describing it is:
[http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/changelog_4_8.html]
Fixed a bug that could cause a queue extent to be recreated if an application deleted a record that was already deleted in that extent.  (#17004)
Added the DB_CONSUME flag to DB->del and DBC->del to force adjustment of the head of the queue. (#17004)
Could you test with BDB 4.8.24 and see if you run into the same problem?
Regards,
Andrei