Frequent stalling and Run Recover messages.

Question

Solaris 9 x86 db-4.2.52.NC-sol9-intel-local
Using named+DLZ in "read only" mode, and local perl script that adds and removes IPs. Very frequently, currently about once a day, the DB will hang (named truss indicates db lock waits perhaps). I can shut everything down, and run recover a few times, and it generally comes back. But it can be somewhat annoying when it happens at 3 am.
Would it possibly help to upgrade to "db-4.6.21.NC" ? After compiling, but not installing, the 4.6.21.NC's db_stat reports "Invalid argument" which makes me suspect I would have to migrate the db first.
I guess I am hoping that either I open it wrong in my script and hence bleed locks or similar. Or, that upgrading to latest version might fix it as there are bugs fixed in that area.
DLZ opens the db with:
if ((result = (*db)->open(*db, NULL, db_file, db_name, db_type, DB_RDONLY | bdbhpt_threads, 0)) != 0) {
My perl script uses:
$env = new BerkeleyDB::Env
-Home => $dbenvpath,
-Flags => DB_INIT_CDB|DB_INIT_MPOOL|DB_CREATE,
-ErrFile => "error.txt",
-Verbose => 1 ;
$dns_data = tie my %cacheDnsData, 'BerkeleyDB::Hash',
-Env => $env,
-Filename => "$dbenvpath/$dbfilename",
-Flags => DB_CREATE,
-Property => DB_DUP | DB_DUPSORT,
-Subname => "dns_data",
or die "Cannot create master.dns_data.db: $BerkeleyDB::Error\n";
tie my %cacheDnsZone, 'BerkeleyDB::Btree',
-Env => $env,
-Filename => "$dbenvpath/$dbfilename",
-Flags => DB_CREATE,
-Subname => "dns_zone",
or die "Cannot create master.dns_zone.db: $BerkeleyDB::Error\n";
Various statistics about the DB (when it is working):
/usr/local/BerkeleyDB.4.2/bin/db_stat -c -h /etc/ns2/
114 Last allocated locker ID.
2147M Current maximum unused locker ID.
5 Number of lock modes.
1000 Maximum number of locks possible.
1000 Maximum number of lockers possible.
1000 Maximum number of lock objects possible.
8 Number of current locks.
19 Maximum number of locks at any one time.
14 Number of current lockers.
22 Maximum number of lockers at any one time.
5 Number of current lock objects.
7 Maximum number of lock objects at any one time.
13M Total number of locks requested.
13M Total number of locks released.
0 Total number of lock requests failing because DB_LOCK_NOWAIT was set.
6 Total number of locks not immediately available due to conflicts.
0 Number of deadlocks.
0 Lock timeout value.
0 Number of locks that have timed out.
0 Transaction timeout value.
0 Number of transactions that have timed out.
400KB The size of the lock region..
137 The number of region locks granted after waiting.
26M The number of region locks granted without waiting.
/usr/local/BerkeleyDB.4.2/bin/db_stat -d /etc/ns/DLZ.dnsdata.db -s dns_data -h /etc/ns2/61561 Hash magic number.8 Hash version number.
Flags: duplicates, multiple-databases, little-endian
8192 Underlying database page size.
0 Specified fill factor.
21M Number of keys in the database.
42M Number of data items in the database.
790653 Number of hash buckets.
2742M Number of bytes free on bucket pages (58% ff).
0 Number of overflow pages.
0 Number of bytes free in overflow pages (0% ff).
46746 Number of bucket overflow pages.
346M Number of bytes free in bucket overflow pages (10% ff).
3413 Number of duplicate pages.
27M Number of bytes free in duplicate pages (2% ff).
0 Number of pages on the free list.
/usr/local/BerkeleyDB.4.2/bin/db_stat -m -h /etc/ns2/
261KB 912B Total cache size.
1 Number of caches.
264KB Pool individual cache size.
0 Requested pages mapped into the process' address space.
14M Requested pages found in the cache (80%).
3823107 Requested pages not found in the cache.
0 Pages created in the cache.
3823136 Pages read into the cache.
18 Pages written from the cache to the backing file.
3823108 Clean pages forced from the cache.
0 Dirty pages forced from the cache.
0 Dirty pages written by trickle-sync thread.
32 Current total page count.
32 Current clean page count.
0 Current dirty page count.
37 Number of hash buckets used for page location.
22M Total number of times hash chains searched for a page.
9 The longest hash chain searched for a page.
29M Total number of hash buckets examined for page location.
49M The number of hash bucket locks granted without waiting.
51 The number of hash bucket locks granted after waiting.
27 The maximum number of times any hash bucket lock was waited for.
11M The number of region locks granted without waiting.
10 The number of region locks granted after waiting.
3823258 The number of page allocations.
8087530 The number of hash buckets examined during allocations
7 The max number of hash buckets examined for an allocation
3823109 The number of pages examined during allocations
1 The max number of pages examined for an allocation
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Pool File: /etc/ns2/DLZ.dnsdata.db
8192 Page size.
0 Requested pages mapped into the process' address space.
14M Requested pages found in the cache (80%).
3823107 Requested pages not found in the cache.
0 Pages created in the cache.
3823136 Pages read into the cache.
18 Pages written from the cache to the backing file.

Answer 1

Thank you for your replies.
I have changed to Transactional, both in the C program (named) and the perl script. I also finally got around to (called too many times at 3am) upgrading BerkeleyDB, so now we are running db-4.6.21.NC.
I have made an automated crontab script that runs every minute, if named do not respond, kill it, kill any perl scripts, and run recover. On an average day, it has to recover some 12-20 times a day.
What I have not quite figured out is, why this situation can occur:
Fri Nov 9 04:02:01 2007 add spammer 65.24.181.238 (cpe-65
-24-181-238.columbus.res.rr.com)
Fri Nov 9 04:07:05 2007 : add spammer 90.193.255.64 (5ac1ff4
0.bb.sky.com)
Fri Nov 09 04:12:00 2007 : FIXING CORRUPT RBL BDB
Fri Nov 9 04:36:14 2007 : add spammer 83.237.203.50 (ppp83-
237-203-50.pppoe.mtu-net.ru)
"named" runs all the time, but definitely opens the DB in read-only. The only other process that runs is the perl script. It added a record at 04:07. Since crontab runs every minute, everything was ok at :08 :09 :10 :11 but at 04:12 it was corrupt. Also notice that in this case, there was no over-lap. Just one writer, and one reader.
Anyway, I can add a test for the DB_LOCK_DEADLOCK error code, but since the "correct action is to close any cursors and abort" and my code only uses cursor with c_get (check for existence of data, or rather, get the second record under same key). All execution of the script is to do one transaction and quit. The "clean" part is not in use yet.
The entire perl script is available here:
http://www.lundman.net/ftp/rbl_add_ip.pl
I could try to upgrade to latest named, but since it opens the DB with READONLY I have assumed the problem must lie with my perl script. Since I recently changed from concurrent to transactional, I do not have any txn_begin() calls, I assume I will have to add them. The examples use db->put, but I assume it will be ok with the perl way of
$cacheDnsData{"$rbldomain $revIP"} ="$serial $revIP $ttl A 127.0.0.$code";
Or is it advisable to drop the db-tie, and use db->func style calls?
Lund