DB_THREAD and set_bt_compare

Hi, All
I set flags DB_THREAD to database before open, and I also use customized compare function. In such situation, should I set DB_DBT_MALLOC both for key and data?
I come cross error "DB_THREAD mandates memory allocation flag on..." but everything is ok is I don't use set_bt_compare before open it.
I can't find clear description in document, so need your clarification, thanks.

Hi,
You do need to specify an explicit memory allocation function on DBTs when using the DB_THREAD flag. You can use any one of the DB_DBT_MALLOC, DB_DBT_USERMEM or DB_DBT_APPMALLOC flags.
If you don't it is possible that the buffer used by Berkeley DB to return results will be overwritten by an operation carried out by another thread, causing unexpected results to be seen by the application.
Regards,
Alex Gorrod
Oracle Berkeley DB

Similar Messages

Set_bt_compare

Hi all,
I want to use a custom sort of secondary duplicates. But I see the FAQ:
[custom sort of secondary duplicates|http://www.oracle.com/technology/products/berkeley-db/faq/db_faq.html#22]
It is not support.
I use the option #1 instead. It work correctly. (I use Btee Access, and set_bt_compare). But there is another problem: Can I use two bt_compare methods for the secondary db? I should support five compostior method !
I use the compare method by numeric the first, but now I want to change it to name. I close the db, and reload it with the new compare method . but it not work.
Should I create five secondary db by five compostior at one time? It waste too much space.
Is there some better options than it?
Thanks,
wendy.

Hello,
Did you set verbose error messages with either:
http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/C/envset_errfile.html
Or if you don't have a database environment handle, with:
http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/C/dbset_errfile.html
The messaging from either of these calls should provide more information on what the problem is.
Thanks,
Sandra

The DbEnv memery missing in win7 x64(may be a berkeley'Env bug in x64)

I am a newer programer in Berkeley,this is my first use it.
I create a BDB for png image, about 40gb, the key is used ACE_UINT64, the value is ACE_Message_Block.
I used LoadRunner create 100 user to get the image by my program.
It is correctly in win7 32bit, but it is lost memory in 64bit.
I open the Env with DB_Private | DB_init_pool | DB_thread, and set the cachesize to 1gb, also the DBt of value is set_flags(DB_DBT_MALLOC), also use free(DBt.getdata()).
My server thread's commit memory in taskmgr.exe is keep at 1gb, but the memory in used of system increase never stop, at last all of memey has been used, and my server thread stop at berkeleydb.
I find my used memory is 8gb, my system+loadruner+vs2008 at most 1.5gb, and my server thread keep in 1gb, what the other memory who used?
So I shut down the server thread, all memory came back.
So I change Berkeley DB Storage to Read my image.png direct in file system, the memory is correctly.
So must some wrong in my code to used berkeleydb, must in DBt’ alloc，so how can i free the memory in x64?
So I need helper， what’s the wrong with my DBEnv？How can I free the DBt in 64 bit？
int IMG_Storage_Environment::Initialize( ISF_Profile_Visitor & Profile )
     Env = new DbEnv( 0 );
     int env_flags = DB_CREATE | // If the environment does not exist, create it
          DB_PRIVATE |
          DB_INIT_MPOOL | // Initialize the cache
          DB_THREAD ; // Free-thread the env handle
     if ( Env->set_cachesize( 1, 0, 1) == 0 &&     Env->open( NULL, env_flags, 0 ) == 0 )
          return ERR_SUCCESS;
int IMG_Storage_BerkeleyDB::Initialize( ACE_StrItrT Layer , ACE_StrItrT Path )
     this->db = new Db( IMG_Storage_Environment::Instance()->getDbEnv(), 0 );
     if (
          0 == db->open( NULL, STR_T2A( Path ) , NULL ,DB_UNKNOWN, DB_RDONLY ,NULL)
          ISF_DEBUG( "Open DB: %s Succeed" , Path );
          return ERR_SUCCESS;
int IMG_Storage_BerkeleyDB::GetTile( int x , int y , int z , ACE_Message_Block & Data )
     ACE_UINT64 uKey=this->Key( x, y, z);
     Dbt dbKey(&uKey, sizeof(uKey));
     Dbt dbData;
     dbData.set_flags( DB_DBT_MALLOC );
     int err = db->get(NULL, & dbKey, & dbData, 0);
     if ( 0 == err )
          Data.size( dbData.get_size( ) );
          Data.rd_ptr( Data.base( ) );
          Data.wr_ptr( dbData.get_size( ) );
          ACE_OS::memcpy( Data.rd_ptr( ) , dbData.get_data( ) , dbData.get_size( ) );
     else
          ISF_DEBUG( "Image Not exist, Using Empty Image" , err );
     free(dbData.get_data());
     return ERR_SUCCESS;
Edited by: 886522 on 2011-9-21 上午1:31
Edited by: 886522 on 2011-9-21 上午1:39

I encounter the same problem, although I run Berkeley DB (Ver 6.0.20, C#) under .NET Framework and Windows server 2008(x64). Any BDB application of win32 runs well but will encounter trouble under platform of x64 when compile BDB to x64, even though the DLL compiled and linked with win32. The bug is that Berkeley DB take amount of memory as the size of databases and regardless of cacheSize. My estimation is that all memory for BDB malloced and NOT freed.

Segmentation fault in __memp_fget

Hi bdb experts,
My program encountered the segfault with the following detailed inf:
# lsb_release -a
LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 5.5 (Tikanga)
Release: 5.5
Codename: Tikanga
filesystem: ext2
gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)
the BDB config:
==================
set_cachesize 2 0 3
env flag: DB_CREATE | DB_INIT_MPOOL | DB_THREAD
db flag: DB_CREATE | DB_THREAD
And my Berkeley database was configured with 3 partitions
as config above, my bdb was configured within a multithread environment
dbp->set_partition(dbp, 3, NULL, db_partition_index)
==================
The coredump back trace:
Program terminated with signal 11, Segmentation fault.
#0 0x00002ad2db2a78b1 in __memp_fget (dbmfp=0x1f9301e0, pgnoaddr=0x46582324, ip=0x0, txn=0x0, flags=0, addrp=0x46582328) at ../src/mp/mp_fget.c:260
260 if (bhp->pgno != *pgnoaddr || bhp->mf_offset != mf_offset)
(gdb) bt
#0 0x00002ad2db2a78b1 in __memp_fget (dbmfp=0x1f9301e0, pgnoaddr=0x46582324, ip=0x0, txn=0x0, flags=0, addrp=0x46582328) at ../src/mp/mp_fget.c:260
#1 0x00002ad2db14adb7 in __bam_search (dbc=0x2aaab89d1430, root_pgno=1, key=0x46582b20, flags=12802, slevel=1, recnop=0x0, exactp=0x465826b4) at ../src/btree/bt_search.c:806
#2 0x00002ad2db1305c4 in __bamc_search (dbc=0x2aaab89d1430, root_pgno=1, key=0x46582b20, flags=14, exactp=0x465826b4) at ../src/btree/bt_cursor.c:2804
#3 0x00002ad2db12e170 in __bamc_put (dbc=0x2aaab89d1430, key=0x46582b20, data=0x46582af0, flags=20, pgnop=0x46582784) at ../src/btree/bt_cursor.c:2143
#4 0x00002ad2db22fd95 in __dbc_iput (dbc=0x2aaab49c68e0, key=0x46582b20, data=0x46582af0, flags=20) at ../src/db/db_cam.c:2134
#5 0x00002ad2db22fbf7 in __dbc_put (dbc=0x2aaab49c68e0, key=0x46582b20, data=0x46582af0, flags=20) at ../src/db/db_cam.c:2047
#6 0x00002ad2db2c6b91 in __partc_put (dbc=0x2aaab5389810, key=0x46582b20, data=0x46582af0, flags=20, pgnop=0x465828b4) at ../src/db/partition.c:1055
#7 0x00002ad2db22fd95 in __dbc_iput (dbc=0x2aaab5389810, key=0x46582b20, data=0x46582af0, flags=20) at ../src/db/db_cam.c:2134
#8 0x00002ad2db22fbf7 in __dbc_put (dbc=0x2aaab5389810, key=0x46582b20, data=0x46582af0, flags=20) at ../src/db/db_cam.c:2047
#9 0x00002ad2db22aad1 in __db_put (dbp=0x1f92db90, ip=0x0, txn=0x0, key=0x46582b20, data=0x46582af0, flags=20) at ../src/db/db_am.c:537
#10 0x00002ad2db24488c in __db_put_pp (dbp=0x1f92db90, txn=0x0, key=0x46582b20, data=0x46582af0, flags=20) at ../src/db/db_iface.c:1640
#11 0x000000000041be46 in bdb::put (this=0x1f92c0a0, key=0x2aaab800103c "Layout:http://emap3.mapabc.com/mapabc/maptile?v=w2.61&x=54&y=26&z=6",
value=0x2aaab8001240 "256|256▒\232-N", vsize=11, ts=1311611596) at backend/bdb.cc:268
#12 0x00000000004151bf in cache_process_add (cmd_no=2, req_head=0x2aaab8001010, req_buf=0x46582e90, res_head=0x2aaacc0008c0, res_buf=0x46582e70) at gate_cache.cpp:1061
#13 0x00000000004121e0 in ub_process_cmdmap (cmd_map=0x508a60, cmd_no=2, req_head=0x2aaab8001010, req_buf=0x46582e90, res_head=0x2aaacc0008c0, res_buf=0x46582e70)
at ../../../../../../public/ub/output/include/ub_proccmd.h:27
#14 0x0000000000414245 in cache_cmdproc_callback () at gate_cache.cpp:1302
#15 0x0000000000469701 in apool_consume (pool=0x1f92e950, data=0x1f92e858) at apool_native.cpp:39
#16 0x000000000044407f in apoolworkers (param=0x1f92e858) at apool.cpp:533
#17 0x00000033b100673d in start_thread () from /lib64/libpthread.so.0
#18 0x00000033b04d3d1d in clone () from /lib64/libc.so.6
My question is:
1) in multithread env bdb put and get method should have some other flag exception DB_THREAD or other config for env and db open?
2) does the ext2 fs may affect the disk read and write for bdb?
3) does the number of threads of my program affect the bdb put and get?
4) what is the suitable page size for 64bit machine of bdb?

Hello,
What is the Berkeley DB version?
For your question on page size please take a look at the
documentation on, "Selecting a page size" at:
http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/general_am_conf.html#am_conf_pagesize
and for those on multithreaded environments please take a
look at the documentation on, "Multithreaded applications" at:
http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/program_mt.html
I am not aware of any impacts of the ext2 filesystem. Perhaps
someone else might have more information on that.
Some other suggestions are to:
0. Build with --enable-diagnostic to enable run-time
debugging checks.
1. turn on verbose error messaging as that often provides
additional run-time error information. Please see the,
"Run-time error information", documentation at:
http://download.oracle.com/docs/cd/E17076_02/html/installation/debug_runtime.html
2. Collect db_stat -MA statistics to to verify that the cache details
look to be in order. See:
http://download.oracle.com/docs/cd/E17076_02/html/api_reference/C/db_stat.html
Thanks,
Sandra

Performance problem using multi-thread

I am using berkeley db to store a int/double pair, and i randomly get 100,0000 results to check the performance . Using 1 thread, it cost about 1 Mins. But using 4 threads, it costs 4 Mins. More threads, lower performance. Is there Anyone know the reason?
Env flags:
envFlags=DB_CREATE | DB_INIT_MPOOL | DB_THREAD | DB_PRIVATE|DB_INIT_CDB;
DB open flags: DB_THREAD
and i use db->get method to get results.

Berkeley DB 4.8 will be released soon and has features that address CMP/SMP scalability. We would appreciate your feedback on your experiences with it. Once available, please test it and post your results here, I'm sure you'll be pleasently surprised.
regards,
-greg

Improving Replicated BDB Performance

Hi all,
I am performing some stress testing on an application which uses Replicated Berkeley DB as an underlying storage system. When running the test I noticed that all the DB servers were mostly idle and waiting and using less than 20% of the CPU. Not surprisingly the bottleneck was the disk which is being heavily written to throughout the test. To improve performance I tried a few of the common Berkely DB settings which are supposed to greatly improve IO performance like DB_TXN_NOSYNC and DB_TXN_WRITE_NOSYNC. Unfortunately adding either of these to the flags had no impact on performance and resulted in the same IO workload. I even tried setting the DB_LOG_IN_MEMORY since that seemed like it would significantly reduce the IO workload, yet still the same result. I find this really odd since these settings seem like they should improve performance and reduce IO, yet they are not. It seems like I may be doing something wrong or have misunderstood the implications of these settings. Also, I made sure to clear out the data directory in-between trials with different settings so make sure nothing weird was happening there.
I am running a replicated Berkeley DB instance (c++) on three separate machines with the following environment flags:
DB_CREATE, DB_RECOVER, DB_INIT_LOCK, DB_INIT_MPOOL, DB_INIT_TXN, DB_INIT_REP, DB_THREAD, and added DB_LOG_IN_MEMORY or DB_TXN_NOSYNC. The only DB flag used is DB_AUTO_COMMIT on a B-Tree database. I am using a 1G cache size which seems to be sufficient because db_stat reports 95%+ hit rates. I am using a 4096 page size since it is the same as that of the file system, along with a 5-50MB log buffer size. I am using the quorum as the ack policy for replicas. In general I have tried to follow as many of the recommendations as possible from the transaction tuning guide.
I am looking to sacrifice some degree of durability for performance, particularly because the machines are running on a local network where an ACK from a replica is likely going to be faster then syncing to disk. Any help in determining how to achieve this would be appreciated.
Edited by: user11122743 on May 29, 2009 11:28 AM

It does seem as if you're on the right track. How much real memory
does your system have, compared to the 1G cache you've configured?
Perhaps your cache is getting swapped out. Have you looked at the
system paging activity statistics?
Alan Bram
Oracle

I encountered 2 error

the code:
u_int32_t id;
u_int32_t dataSize;
char dataBuf[32];
dataSize = getData(dataBuf,..);
Dbt key(&id, sizeof(id));
key.set_flag(DB_DBT_USERMEM);
key.set_ulen(sizeof(id));
Dbt data(buffer, buf_size);
Dbc* cursorp;
db.cursor(NULL, &cursorp, 0);
int ret = cursorp->get(&key, &data, DB_GET_BOTH);
if (0 == ret) {
// here I saw dataBuf had been changed, it was be filled with the data in db
// but I had not called data.set_flag(DB_DBT_USERMEM), was it a bug? or it was the design?
// my dup_compare only compare part of data
ret = cursor.del(0); assert(0 == ret);
// after I found the error, I call my getData(...) again here:
newID = updateID(....);
dataSize = getData(dataBuf,..);
data.set_data(dataBuf);
data.set_size(dataSize);
key.set_data(&newID);
ret = db.put(NULL, &key, &data, 0); assert(0 == ret);
ret = cursorp->close(); assert(0 == ret);
another error, when i call db.sync(0), I get a message:
memp_sync: correcting dirty count 7 4
I used multi different Db in one DbEnv, and every Db was used in different thread, DbEnv was opened with DB_THREAD, and Db was opened without DB_THREAD. when muti thread was joined, I call every db.sync(0), then I saw this message.

Hi,
Could you post a ready to be run small reproducible testcase that demonstrate these issues? If so, mention the Berkeley DB release you're using and on what platform/OS.
Regards,
Andrei

Configuration for best performance with concurrent access

Could someone tell me the flags I need to specify to achieve the best BDB performance in a multithreaded (concurrent) environment?
I am using Hash type and have simple access protocol. Just key/value get/put and delete. I don't need transactions or recovery
I tried DB_CREATE | DB_INIT_MPOOL | DB_INIT_CDB | DB_THREAD env. open flags and DB_CREATE | DB_THREAD db open flags, but I can only achieve 10000 requests per second.
I can get up to 40000 RPS with a single thread and just DB_CREATE | DB_INIT_MPOOL for environment and DB_CREATE for database. That should be sufficient, but I need that performance in multithreaded environment.
I tried EnvOpenFlags = DB_CREATE | DB_INIT_MPOOL | DB_THREAD and DbOpenFlags = DB_CREATE | DB_THREAD, but I am getting the following errors on Put:
page 29778: illegal page type or format
PANIC: fatal region error detected; run recovery
PANIC: Invalid argument
If I add DB_INIT_LOCK to the EnvOpenFlags, I get deadlocks. I am not sure why I even getting them based on my access schema.
Thanks,
Igor

I am running test on 64 bit Windows 2003.
Data is accessed randomly, but in the test, I do only inserts. Data is just a list of key/value pairs. Key is int and Value is byte array or size 50 (but it could vary up to about 50000).
In production it could be around 30 database files (one database per file), but for the test I use only one file.
I don't have config.log file. I built only db_dll and db_stat projects in VSN2005 and I use it form C# code through managed C++ wrapper.
Here are the stats for the DB_CREATE | DB_INIT_MPOOL | DB_INIT_CDB | DB_THREAD set of flags:
db_stat -e =>
Tue Feb 06 09:44:37 2007 Local time
0x120897 Magic number
0 Panic value
4.5.20 Environment version
Mon Feb 05 17:57:21 2007 Creation time
0x440a8ca8 Environment ID
1 Primary region allocation and reference count mutex [0/0 0% !Own]
2 References
Thread status blocks:
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
33 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
5 Number of lock modes
1000 Maximum number of locks possible
1000 Maximum number of lockers possible
1000 Maximum number of lock objects possible
3 Number of current locks
5 Maximum number of locks at any one time
4 Number of current lockers
6 Maximum number of lockers at any one time
2 Number of current lock objects
4 Maximum number of lock objects at any one time
6813619 Total number of locks requested
6813616 Total number of locks released
0 Total number of locks upgraded
12 Total number of locks downgraded
3262547 Lock requests not available due to conflicts, for which we waited
0 Lock requests not available due to conflicts, for which we did not wait
0 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
536KB The size of the lock region
8090 The number of region locks that required waiting (0%)
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
5GB 50MB Total cache size
1 Number of caches
5GB 50MB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
17M Requested pages found in the cache (99%)
25834 Requested pages not found in the cache
22234 Pages created in the cache
11856 Pages read into the cache
88248 Pages written from the cache to the backing file
0 Clean pages forced from the cache
0 Dirty pages forced from the cache
0 Dirty pages written by trickle-sync thread
34063 Current total page count
5424 Current clean page count
28639 Current dirty page count
786431 Number of hash buckets used for page location
17M Total number of times hash chains searched for a page (17031608)
1 The longest hash chain searched for a page
16M Total number of hash chain entries checked for page (16971686)
0 The number of hash bucket locks that required waiting (0%)
0 The maximum number of times any hash bucket lock was waited for (0%)
0 The number of region locks that required waiting (0%)
0 The number of buffers frozen
0 The number of buffers thawed
0 The number of frozen buffers freed
34100 The number of page allocations
0 The number of hash buckets examined during allocations
0 The maximum number of hash buckets examined for an allocation
0 The number of pages examined during allocations
0 The max number of pages examined for an allocation
0 Threads waited on page I/O
Pool File: data.bdb30
8192 Page size
0 Requested pages mapped into the process' address space
17M Requested pages found in the cache (99%)
25832 Requested pages not found in the cache
22234 Pages created in the cache
11854 Pages read into the cache
88240 Pages written from the cache to the backing file
Pool File: e:/BerkeleyDb\admin.bdb
8192 Page size
0 Requested pages mapped into the process' address space
19 Requested pages found in the cache (90%)
2 Requested pages not found in the cache
0 Pages created in the cache
2 Pages read into the cache
8 Pages written from the cache to the backing file
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
96MB 88KB Mutex region size
0 The number of region locks that required waiting (0%)
4 Mutex alignment
100 Mutex test-and-set spins
1574134 Mutex total count
1239 Mutex free count
1572895 Mutex in-use count
1572897 Mutex maximum in-use count
Mutex counts
1239 Unallocated
1 db handle
1 env dblist
1 env region
1 lock region
5 logical lock
2 mpoolfile handle
1 mpool filehandle
17 mpool file bucket
1 mpool handle
786431 mpool hash bucket
786431 mpool buffer I/O
1 mpool region
1 unknown mutex type
1 twister
db_stat -d =>
Tue Feb 06 09:46:42 2007 Local time
61561 Hash magic number
8 Hash version number
Little-endian Byte order
Flags
8192 Underlying database page size
0 Specified fill factor
6048411 Number of keys in the database
6048411 Number of data items in the database
27229 Number of hash buckets
14M Number of bytes free on bucket pages (93% ff)
0 Number of overflow pages
0 Number of bytes free in overflow pages (0% ff)
8438 Number of bucket overflow pages
30M Number of bytes free in bucket overflow pages (57% ff)
0 Number of duplicate pages
0 Number of bytes free in duplicate pages (0% ff)
1 Number of pages on the free list
db_stat -m =>
5GB 50MB Total cache size
1 Number of caches
5GB 50MB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
17M Requested pages found in the cache (99%)
27219 Requested pages not found in the cache
22444 Pages created in the cache
13241 Pages read into the cache
120831 Pages written from the cache to the backing file
0 Clean pages forced from the cache
0 Dirty pages forced from the cache
0 Dirty pages written by trickle-sync thread
35672 Current total page count
35658 Current clean page count
14 Current dirty page count
786431 Number of hash buckets used for page location
17M Total number of times hash chains searched for a page (17471571)
1 The longest hash chain searched for a page
17M Total number of hash chain entries checked for page (17408653)
0 The number of hash bucket locks that required waiting (0%)
0 The maximum number of times any hash bucket lock was waited for (0%)
0 The number of region locks that required waiting (0%)
0 The number of buffers frozen
0 The number of buffers thawed
0 The number of frozen buffers freed
35695 The number of page allocations
0 The number of hash buckets examined during allocations
0 The maximum number of hash buckets examined for an allocation
0 The number of pages examined during allocations
0 The max number of pages examined for an allocation
0 Threads waited on page I/O
Pool File: data.bdb30
8192 Page size
0 Requested pages mapped into the process' address space
17M Requested pages found in the cache (99%)
27217 Requested pages not found in the cache
22444 Pages created in the cache
13239 Pages read into the cache
120821 Pages written from the cache to the backing file
Pool File: e:/BerkeleyDb\admin.bdb
8192 Page size
0 Requested pages mapped into the process' address space
25 Requested pages found in the cache (92%)
2 Requested pages not found in the cache
0 Pages created in the cache
2 Pages read into the cache
10 Pages written from the cache to the backing file
db_stat -CA =>
Default locking region information:
39 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
5 Number of lock modes
1000 Maximum number of locks possible
1000 Maximum number of lockers possible
1000 Maximum number of lock objects possible
0 Number of current locks
5 Maximum number of locks at any one time
0 Number of current lockers
6 Maximum number of lockers at any one time
0 Number of current lock objects
4 Maximum number of lock objects at any one time
6975306 Total number of locks requested
6975306 Total number of locks released
0 Total number of locks upgraded
15 Total number of locks downgraded
3423839 Lock requests not available due to conflicts, for which we waited
0 Lock requests not available due to conflicts, for which we did not wait
0 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
536KB The size of the lock region
8274 The number of region locks that required waiting (0%)
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock REGINFO information:
Lock Region type
4 Region ID
e:\BerkeleyDB\__db.004 Region name
0x90a0000 Original region address
0x90a0000 Region address
0x9125f00 Region primary address
0 Region maximum allocation
0 Region allocated
REGION_JOIN_OK Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock region parameters:
1572884 Lock region region mutex [8274/17M 0% 1552/2884]
1031 locker table size
1031 object table size
532064 obj_off
515560 locker_off
1 need_dd
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock conflict matrix:
0 0 0 0 0
0 0 1 0 0
0 1 1 1 1
0 0 0 0 0
0 0 1 0 1
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by lockers:
Locker Mode Count Status ----------------- Object ---------------
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by object:
Locker Mode Count Status ----------------- Object ---------------

Environment DB_PRIVATE flag

Hello
I've got a question: Is it possible to open several process-private environments that would share their regions, i.e. would have the same effect as opening the environment in some directory but would not create any files?
And one more question: Do I have to enable locking if I have concurrent accesses to the environment, but not to the underlying databases. I mean, is access to the memory pool and other environment regions already synchronized, without specifying DB_INIT_LOCK?
Thanks, Giorgi.
Edited by: _Hitman47 on Dec 10, 2010 1:53 AM

Yeah, I've read documentation. My question was following: Can I open several private environment handles per process (one per thread) which would use the same memory pool? I can make a private environment handle thread safe by specifying DB_THREAD flag and use that single handle in all threads, but performance overhead of using DB_THREAD flag is quite big - about 10%.
My idea is to use one environment handle per thread, with only DB_THREAD and DB_PRIVATE flags specified, but those environment handles MUST use the same memory pool.
In my application, each thread opens a set of databases, do some operations on them and close them. My application guarantees that no database will be opened simultaneously by more than one thread at one moment of time.
The problems begin when one thread opens a database in its own private environment, which has been previously opened(and closed!) by another thread. It seems that closing database in environment still leaves some pages unflushed to hard disk, even after calling DB_ENV->memp_sync() function. So when another thread tries to load that database, ot is not consistent and errors start to appear. If I use one single private environment handle for all threads, or open nonprivate(file-backed) environment handles in each thread, the application works as it should.
I hope I expressed myself clear enough.
Thanks, Giorgi.

Lock table is out of available object entries

hi,
i am using db4.6.21 version.
i have created an table where other applications writes to this tables concurrently.
this table opened in DB_THREAD and every application writing in DB_WRITE_CURSOR mode. and i am note using any Locking subsystems.only READ_COMMITTED and DB_WRITE_CURSOR are used by applications to access the table.
while In PC it is working properly.
But in At91SAM9260EK with kerenl 2.6.23.9 ,
Berkeley DB error: Lock table is out of available object entries
error comes...what would be the reason..???

Hi Ratheesh,
Please search through the forum; similar locking subsystem configuration issues have already been discussed.
In short, you'll need to increase the number of lock objects:
http://www.oracle.com/technology/documentation/berkeley-db/db/ref/lock/max.html
I see you're using the DB_WRITECURSOR flag which is specific to CDS (Concurrent Data Store), so you should size the locking subsystem appropriately to CDS: the number of lock objects needed is two per open database (one for the database lock, and one for the cursor lock when the DB_CDB_ALLDB option is not specified). The locking subsystem configuration should be similar for all the processes accessing the environment, or not specified for the processes that just join the environment.
If you still see this error message reported, provide some information on your OS/platform, information on how the processes access the environment and the locking statistics (db_stat -N -Co -h <env_dir>).
Regards,
Andrei

Need help with sorting records in primary and secondary databases

Hi,
I would like to store data into primary and secondary db in different order. For the main primary, I want it to be ordered by login_ts instead of uuid which is the key.
For the user secondary database, I want it to be ordered by sip_user. For the timestampe secondary db, I want it to be ordered by login_ts.
This is what I have right now,
this is for main
uuid=029ae227-a188-4ba8-aea4-7cbc26783d6 sip_user=200003 login_ts=1264327630 logout_ts=
uuid=22966f76-8c8a-4ab4-b832-b36e8f8e14d sip_user=200003 login_ts=1264327688 logout_ts=
uuid=e1846e4a-e1f5-406d-b903-55905a2533a sip_user=200003 login_ts=1264327618 logout_ts=
uuid=e2f9a3cb-02d1-47ff-8af8-a3a371e20b5 sip_user=200003 login_ts=1264327613 logout_ts=
this is for user search
uuid=029ae227-a188-4ba8-aea4-7cbc26783d6 sip_user=200003 login_ts=1264327630 logout_ts=
uuid=22966f76-8c8a-4ab4-b832-b36e8f8e14d sip_user=200003 login_ts=1264327688 logout_ts=
uuid=e1846e4a-e1f5-406d-b903-55905a2533a sip_user=200003 login_ts=1264327618 logout_ts=
uuid=e2f9a3cb-02d1-47ff-8af8-a3a371e20b5 sip_user=200003 login_ts=1264327613 logout_ts=
this is for timestamp
uuid=029ae227-a188-4ba8-aea4-7cbc26783d6 sip_user=200003 login_ts=1264327630 logout_ts=
uuid=22966f76-8c8a-4ab4-b832-b36e8f8e14d sip_user=200003 login_ts=1264327688 logout_ts=
uuid=e1846e4a-e1f5-406d-b903-55905a2533a sip_user=200003 login_ts=1264327618 logout_ts=
uuid=e2f9a3cb-02d1-47ff-8af8-a3a371e20b5 sip_user=200003 login_ts=1264327613 logout_ts=
but what I want is :
this is for main
uuid=e2f9a3cb-02d1-47ff-8af8-a3a371e20b5 sip_user=200003 login_ts=1264327613 logout_ts=
uuid=e1846e4a-e1f5-406d-b903-55905a2533a sip_user=200004 login_ts=1264327618 logout_ts=
uuid=029ae227-a188-4ba8-aea4-7cbc26783d6 sip_user=200003 login_ts=1264327630 logout_ts=
uuid=22966f76-8c8a-4ab4-b832-b36e8f8e14d sip_user=200005 login_ts=1264327688 logout_ts=
this is for user search
uuid=e2f9a3cb-02d1-47ff-8af8-a3a371e20b5 sip_user=200003 login_ts=1264327613 logout_ts=
uuid=029ae227-a188-4ba8-aea4-7cbc26783d6 sip_user=200003 login_ts=1264327630 logout_ts=
uuid=e1846e4a-e1f5-406d-b903-55905a2533a sip_user=200004 login_ts=1264327618 logout_ts=
uuid=22966f76-8c8a-4ab4-b832-b36e8f8e14d sip_user=200004 login_ts=1264327688 logout_ts=
this is for timestamp
uuid=e2f9a3cb-02d1-47ff-8af8-a3a371e20b5 sip_user=200003 login_ts=1264327613 logout_ts=
uuid=e1846e4a-e1f5-406d-b903-55905a2533a sip_user=200003 login_ts=1264327618 logout_ts=
uuid=029ae227-a188-4ba8-aea4-7cbc26783d6 sip_user=200004 login_ts=1264327630 logout_ts=
uuid=22966f76-8c8a-4ab4-b832-b36e8f8e14d sip_user=200004 login_ts=1264327688 logout_ts=
Right now, I have:
int compare_login_ts(dbp, a, b)
     DB *dbp;
     const DBT a, b;
     int time_a = 0;
     int time_b = 0;
     time_a = atoi ( (char *)a->data);
     time_b = atoi ( (char *)b->data);
     return time_a - time_b ;
for the timestamp secondary, I set that compare function:
          if ((ret = (*sdb)->set_bt_compare(*sdb , compare_login_ts )) != 0){
Does anyone know how can I make it sorted according?

Hi,
The DB->set_bt_compare() is used to compare keys in Btree database. In the callback function, both the DBTs are key, but not data. Please refer to http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/C/dbset_bt_compare.html.
If you want any field in the data to be sorted, you might create a secondary index on it and define the compare function as you wish.
Regards,
Emily Fu, Oracle Berkeley DB

How to configure ENV and DB for multithreaded application?

Hi,
From document, I know DB_THREAD must be checked for both ENV and DB, but , I don't know which one is best choice for multithreaded application while facing DB_INIT_LOCK and DB_INIT_CDB. In my application, there maybe multi readers and writers at the same time, should I use DB_INIT_LOCK instead of DB_INIT_CDB? what other flags should I use?
DB_INIT_CDB provides multiple reader/single writer access while DB_INIT_LOCK should be used when multiple processes or threads are going to be reading and writing a Berkeley DB database.
Thanks for your seggestions and answers.

Thanks for the explanation,
The Berkeley DB Concurrent Data Store product
allows for multiple reader/single writer access
to a database. This means that at any point in time,
there may be either multiple readers accessing a
database or a single writer updating the database.
Berkeley DB Concurrent Data Store is intended for
applications that need support for concurrent updates
to a database that is largely used for reading.
If you are looking to support multiple readers and
multiple writers then take a look at the Transactional
Data Store product
(http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/transapp.html)
In this case the Environment is typically opened with:
DB_INIT_MPOOL, DB_INIT_LOCK, DB_INIT_LOG, and DB_INIT_TXN.
Let me know if I missed any of your question.
Thanks,
Sandra

Log file corrupt and can't open the database.

I use Replication Manager to manage my Replication; and there is only one site currently in my application.
I killed the App process with signal -9. (ie. $ kill -9 appID).
Then I try to restart the App, but the event_callback function got a PANIC event when to open the envirment.
The open flag is:
flags = DB_CREATE | DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL |
DB_INIT_REP | DB_INIT_TXN | DB_RECOVER | DB_THREAD;
ret = dbenv->open(dbenv, path, flags | DB_SYSTEM_MEM, 0);
What's the reason cause this problem?
How can recover it ?
The logs list as below:
[D 11/19 09:44] dbpf using shm key: 977431433
[E 11/19 09:44] [src/io/dbpf-mgmt.c, 400]
[E 11/19 09:44] yuga: DB_LOGC->get: LSN 1/9776: invalid log record header
[E 11/19 09:44] yuga: Log file corrupt at LSN: [1][9906]
[E 11/19 09:44] yuga: PANIC: Invalid argument
[E 11/19 09:44] [src/io/dbpf-mgmt.c] Rep EventGot a panic: Invalid argument (22)
Edited by: dbnicker on Nov 18, 2010 6:08 PM

First, what version of Berkeley DB are you running and on what system?
The error indicates something amiss in the log. The LSN values are quite
small. Can you run 'db_printlog -N -h <env path>' and post the log
contents?
If you are using BDB 5.0 or later, can you also post the contents of
the __db.rep.diag00 file in the environment home directory? Thanks.
Sue LoVerso
Oracle

Environment open flags and multithreading

Do I actually need to create an environment explicitly if I want to read only from a database in a multithreaded program (no other process, no write operations)? I don't need to use DB_INIT_LOCK as there is no thread writing to the database. And I don't need to use DB_INIT_MPOOL as there is no other process using the very same database. As the database is read-only the open flags DB_INIT_LOG, DB_INIT_TXN and DB_RECOVER don't provide any benefit either. Is this all correct? I don't need to use DB_THREAD either (which I found in the documentation of DbEnv)?
Boris

LaurenFoutz wrote:
You are correct. Using XmlManager without an environment will automatically create an environment that can support multithreaded access (but not multiprocess). So for your multithreaded single-process read only application an XmlManager without an explicit environment should be sufficient.I started to test some code. Unfortunately it always crashs if I use more than one thread. I get different error messages when it crashs - here are a few:
{font:Courier}BDB XML: page 0: illegal page type or format
BDB XML: PANIC: Invalid argument
BDB XML: assert failure: ..\..\db-4.6.21\db\db_cam.c/92: "F_ISSET(dbc, DBC_ACTIVE)"
BDB XML: PANIC: fatal region error detected; run recovery{font}
{font:Courier}BDB XML: assert failure: ..\..\db-4.6.21\mp\mp_alloc.c/564: "(bhp == first_bhp)? priority == last_priority : priority >= last_priority"{font}
{font:Courier}BDB XML: test.bdbxml: more pages returned than retrieved
BDB XML: PANIC: Permission denied
BDB XML: PANIC: fatal region error detected; run recovery{font}
I still have to debug the code to find out if I am doing anything wrong. But are there any known issues that it makes sense if I try to use some flags and maybe create the environment explicitly? Currently I only use DB_RDONLY to open the container.
Boris
PS: I just saw that the API reference contains a flag DB_THREAD. If I open the container with DB_THREAD though I get another error message:
{font:Courier}BDB XML: environment not created using DB_THREAD{font}
I'm a bit confused now: Either DB_THREAD is not required for thread-safe access or an environment must be created explicitly in order to use DB_THREAD?
Edited by: Boris Schaeling on Jul 31, 2009 2:11 AM

MPOOL, mmap and linux

I have two processes that open and use the same database (one reads, one writes). The DBENV is opened with the following flags: DB_CREATE | DB_INIT_CDB | DV_INIT_MPOOL | DB_THREAD. The database itself is about currently 1G in size and contains 5M records. I am setting the DBENV cache size to be 2.5G.
As linux will mmap() the database file, what does setting a DBENV cache buy me? Can't I just use the OS buffer cache to take care of my DB pages?
Thanks for any help
Ashley
OS: centos5.0 x86_64
Kernel: 2.6.18
Ram 4G
db_stat -m output:
2GB 512MB Total cache size
1 Number of caches
2GB 512MB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
1344M Requested pages found in the cache (99%)
2 Requested pages not found in the cache
259317 Pages created in the cache
2 Pages read into the cache
2 Pages written from the cache to the backing file
0 Clean pages forced from the cache
0 Dirty pages forced from the cache
0 Dirty pages written by trickle-sync thread
259319 Current total page count
0 Current clean page count
259319 Current dirty page count
262147 Number of hash buckets used for page location
1344M Total number of times hash chains searched for a page (1344270607)
1 The longest hash chain searched for a page
1344M Total number of hash buckets examined for page location (1344013832)
636 The number of hash bucket locks that required waiting (0%)
529 The maximum number of times any hash bucket lock was waited for
0 The number of region locks that required waiting (0%)
259324 The number of page allocations
0 The number of hash buckets examined during allocations
0 The maximum number of hash buckets examined for an allocation
0 The number of pages examined during allocations
0 The max number of pages examined for an allocation

Hello,
The "Configuring the memory pool" section of the Reference Guide at:
http://www.oracle.com/technology/documentation/berkeley-db/db/ref/mp/config.html
has the following:
"Mapping files into the process address space can result in better performance
because available virtual memory is often much larger than the local cache,
and page faults are faster than page copying on many systems. However, in the
presence of limited virtual memory, it can cause resource starvation; and in
the presence of large databases, it can result in immense process sizes. In
addition, because of the requirements of the Berkeley DB transactional
implementation, only read-only files can be mapped into process memory."
One consideration here is the maximum size an underlying file can be and still
be mapped into the process address space (instead of reading the file's pages
into the cache).
I am not sure that this answers your question, so if it does not, please
let me know.
Thanks,
Sandra

DB_THREAD and set_bt_compare

Similar Messages

Maybe you are looking for