BDB dumps core after adding approx 19MB of data

Question

Hi,
BDB core dumps after adding about 19MB of data & killing and restarting it several times.
Stack trace :
#0 0xc00000000033cad0:0 in kill+0x30 () from /usr/lib/hpux64/libc.so.1
(gdb) bt
#0 0xc00000000033cad0:0 in kill+0x30 () from /usr/lib/hpux64/libc.so.1
#1 0xc000000000260cf0:0 in raise+0x30 () from /usr/lib/hpux64/libc.so.1
#2 0xc0000000002fe710:0 in abort+0x190 () from /usr/lib/hpux64/libc.so.1
warning:
ERROR: Use the "objectdir" command to specify the search
path for objectfile db_err.o.
If NOT specified will behave as a non -g compiled binary.
warning: No unwind information found.
Skipping this library /integhome/jobin/B063_runEnv/add-ons/lib/libicudata.sl.34.
#3 0xc000000022ec2340:0 in __db_assert+0xc0 ()
from /integhome/jobin/B063_runEnv/service/sys/servicerun/bin/libdb_cxx-4.3.so
warning:
ERROR: Use the "objectdir" command to specify the search
path for objectfile db_meta.o.
If NOT specified will behave as a non -g compiled binary.
#4 0xc000000022ed2870:0 in __db_new+0x780 ()
from /integhome/jobin/B063_runEnv/service/sys/servicerun/bin/libdb_cxx-4.3.so
warning:
ERROR: Use the "objectdir" command to specify the search
path for objectfile bt_split.o.
If NOT specified will behave as a non -g compiled binary.
#5 0xc000000022ded690:0 in __bam_root+0xb0 ()
from /integhome/jobin/B063_runEnv/service/sys/servicerun/bin/libdb_cxx-4.3.so
#6 0xc000000022ded2d0:0 in __bam_split+0x1e0 ()
from /integhome/jobin/B063_runEnv/service/sys/servicerun/bin/libdb_cxx-4.3.so
warning:
ERROR: Use the "objectdir" command to specify the search
path for objectfile bt_cursor.o.
If NOT specified will behave as a non -g compiled binary.
#7 0xc000000022dc83f0:0 in __bam_c_put+0x360 ()
from /integhome/jobin/B063_runEnv/service/sys/servicerun/bin/libdb_cxx-4.3.so
warning:
ERROR: Use the "objectdir" command to specify the search
path for objectfile db_cam.o.
If NOT specified will behave as a non -g compiled binary.
#8 0xc000000022eb8c10:0 in __db_c_put+0x740 ()
from /integhome/jobin/B063_runEnv/service/sys/servicerun/bin/libdb_cxx-4.3.so
warning:
ERROR: Use the "objectdir" command to specify the search
path for objectfile db_am.o.
If NOT specified will behave as a non -g compiled binary.
#9 0xc000000022ea4100:0 in __db_put+0x4c0 ()
from /integhome/jobin/B063_runEnv/service/sys/servicerun/bin/libdb_cxx-4.3.so---Type <return> to continue, or q <return> to quit---
warning:
ERROR: Use the "objectdir" command to specify the search
path for objectfile db_iface.o.
If NOT specified will behave as a non -g compiled binary.
#10 0xc000000022eca7a0:0 in __db_put_pp+0x240 ()
from /integhome/jobin/B063_runEnv/service/sys/servicerun/bin/libdb_cxx-4.3.so
warning:
ERROR: Use the "objectdir" command to specify the search
path for objectfile cxx_db.o.
If NOT specified will behave as a non -g compiled binary.
#11 0xc000000022d92c90:0 in Db::put(DbTxn*,Dbt*,Dbt*,unsigned int)+0x120 ()
from /integhome/jobin/B063_runEnv/service/sys/servicerun/bin/libdb_cxx-4.3.so
What is the behaviour of BDB if its killed & restarted when a bdb transaction is in progress?
anybody has an idea as to why BDB dumps core in above scenario?
Regards
Sandhya

Answer 1

Hi Bogdan,
As suggested by you i am using the below flags to open an enviornment.
DB_RECOVER |DB_CREATE | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN|DB_THREAD
DB_INIT_LOCK is not used because at our application level we are maintaining a lock to guard against multiple simultaneous access.
The foll msg is output on the console & the dumps core with same stack trace as posted before.
__db_assert: "last == pgno" failed: file "../dist/../db/db_meta.c", line 163
I ran db_verify, db_stat, db_recover tools on the DB & thier results are as below.
db_verify <dbfile>
db_verify: Page 4965: partially zeroed page
db_verify: ./configserviceDB: DB_VERIFY_BAD: Database verification failed
db_recover -v
Finding last valid log LSN: file: 1 offset 42872
Recovery starting from [1][42200]
Recovery complete at Sat Jul 28 17:40:36 2007
Maximum transaction ID 8000000b Recovery checkpoint [1][42964]
db_stat -d <dbfile>
53162 Btree magic number
9 Btree version number
Big-endian Byte order
Flags
2 Minimum keys per-page
8192 Underlying database page size
1 Number of levels in the tree
60 Number of unique keys in the tree
60 Number of data items in the tree
0 Number of tree internal pages
0 Number of bytes free in tree internal pages (0% ff)
1 Number of tree leaf pages
62 Number of bytes free in tree leaf pages (99% ff)
0 Number of tree duplicate pages
0 Number of bytes free in tree duplicate pages (0% ff)
0 Number of tree overflow pages
0 Number of bytes free in tree overflow pages (0% ff)
0 Number of empty pages
0 Number of pages on the free list
db_stat -E <dbfile>
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Default database environment information:
4.3.28 Environment version
0x120897 Magic number
0 Panic value
2 References
0 The number of region locks that required waiting (0%)
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Per region database environment information:
Mpool Region:
2 Region ID
-1 Segment ID
1MB 264KB Size
0 The number of region locks that required waiting (0%)
Log Region:
3 Region ID
-1 Segment ID
1MB 64KB Size
0 The number of region locks that required waiting (0%)
Transaction Region:
4 Region ID
-1 Segment ID
16KB Size
0 The number of region locks that required waiting (0%)
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
DB_ENV handle information:
Set Errfile
db_stat Errpfx
!Set Errcall
!Set Feedback
!Set Panic
!Set Malloc
!Set Realloc
!Set Free
Verbose flags
!Set App private
!Set App dispatch
!Set Home
!Set Log dir
/integhome/jobin/B064_July2/runEnv/temp Tmp dir
!Set Data dir
0660 Mode
DB_INIT_LOG, DB_INIT_MPOOL, DB_INIT_TXN, DB_USE_ENVIRON Open flags
!Set Lockfhp
Set Rec tab
187 Rec tab slots
!Set RPC client
0 RPC client ID
0 DB ref count
-1 Shared mem key
400 test-and-set spin configuration
!Set DB handle mutex
!Set api1 internal
!Set api2 internal
!Set password
!Set crypto handle
!Set MT mutex
DB_ENV_LOG_AUTOREMOVE, DB_ENV_OPEN_CALLED Flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Default logging region information:
0x40988 Log magic number
10 Log version number
1MB Log record cache size
0660 Log file mode
1Mb Current log file size
632B Log bytes written
632B Log bytes written since last checkpoint
1 Total log file writes
0 Total log file write due to overflow
1 Total log file flushes
1 Current log file number
42872 Current log file offset
1 On-disk log file number
42872 On-disk log file offset
1 Maximum commits in a log flush
1 Minimum commits in a log flush
1MB 64KB Log region size
0 The number of region locks that required waiting (0%)
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Log REGINFO information:
Log Region type
3 Region ID
__db.003 Region name
0xc00000000b774000 Original region address
0xc00000000b774000 Region address
0xc00000000b883dd0 Region primary address
0 Region maximum allocation
0 Region allocated
REGION_JOIN_OK Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
DB_LOG handle information:
!Set DB_LOG handle mutex
0 Log file name
!Set Log file handle
Flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
LOG handle information:
0 file name list mutex (0%)
0x40988 persist.magic
10 persist.version
0 persist.log_size
0660 persist.mode
1/42872 current file offset LSN
1/42872 first buffer byte LSN
0 current buffer offset
42872 current file write offset
68 length of last record
0 log flush in progress
0 Log flush mutex (0%)
1/42872 last sync LSN
1/41475 cached checkpoint LSN
1MB log buffer size
1MB log file size
1MB next log file size
0 transactions waiting to commit
1/0 LSN of first commit
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
LOG FNAME list:
0 File name mutex (0%)
1 Fid max
ID Name Type Pgno Txnid DBP-info
0 configserviceDB btree 0 0 No DBP 0 0 0
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Default cache region information:
1MB 262KB 960B Total cache size
1 Number of caches
1MB 264KB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
43312 Requested pages found in the cache (89%)
4968 Requested pages not found in the cache
640 Pages created in the cache
4965 Pages read into the cache
621 Pages written from the cache to the backing file
4818 Clean pages forced from the cache
621 Dirty pages forced from the cache
0 Dirty pages written by trickle-sync thread
166 Current total page count
146 Current clean page count
20 Current dirty page count
131 Number of hash buckets used for page location
53888 Total number of times hash chains searched for a page
4 The longest hash chain searched for a page
92783 Total number of hash buckets examined for page location
0 The number of hash bucket locks that required waiting (0%)
0 The maximum number of times any hash bucket lock was waited for
0 The number of region locks that required waiting (0%)
5615 The number of page allocations
10931 The number of hash buckets examined during allocations
22 The maximum number of hash buckets examined for an allocation
5439 The number of pages examined during allocations
11 The max number of pages examined for an allocation
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Pool File: temporary
1024 Page size
0 Requested pages mapped into the process' address space
43245 Requested pages found in the cache (99%)
1 Requested pages not found in the cache
635 Pages created in the cache
0 Pages read into the cache
617 Pages written from the cache to the backing file
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Pool File: configserviceDB
8192 Page size
0 Requested pages mapped into the process' address space
65 Requested pages found in the cache (1%)
4965 Requested pages not found in the cache
1 Pages created in the cache
4965 Pages read into the cache
0 Pages written from the cache to the backing file
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Mpool REGINFO information:
Mpool Region type
2 Region ID
__db.002 Region name
0xc00000000b632000 Original region address
0xc00000000b632000 Region address
0xc00000000b773f08 Region primary address
0 Region maximum allocation
0 Region allocated
REGION_JOIN_OK Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
MPOOL structure:
0/0 Maximum checkpoint LSN
131 Hash table entries
64 Hash table last-checked
48905 Hash table LRU count
48914 Put counter
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
DB_MPOOL handle information:
!Set DB_MPOOL handle mutex
1 Underlying cache regions
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
DB_MPOOLFILE structures:
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
MPOOLFILE structures:
File #1: temporary
0 Mutex (0%)
0 Reference count
18 Block count
634 Last page number
0 Original last page number
0 Maximum page number
0 Type
0 Priority
0 Page's LSN offset
32 Page's clear length
0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 f8 0 0 0 0 ID
deadfile, file written Flags
File #2: configserviceDB
0 Mutex (0%)
1 Reference count
148 Block count
4965 Last page number
4964 Original last page number
0 Maximum page number
0 Type
0 Priority
0 Page's LSN offset
32 Page's clear length
0 0 b6 59 40 1 0 2 39 ac 13 6f 0 a df 18 0 0 0 0 ID
file written Flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Cache #1:
BH hash table (131 hash slots)
bucket #: priority, mutex
pageno, file, ref, LSN, mutex, address, priority, flags
bucket 0: 47385, 0/0%:
4813, #2, 0, 0/1, 0/0%, 0x04acf0, 47385
4944, #2, 0, 0/0, 0/0%, 0x020c18, 48692