Getting `No such file or directory` error while trying to open bdb database

Question

I have four multi-threaded processes (2 writer and 2 reader processes), which make use of Berkeley DB transactional data store. I have multiple environments and the associated database files and log files are located in separate directories (please refer to the DB_CONFIG below). When all these four processes start to perform open and close of databases in the environments very quickly, one of the reader process is throwing a No such file or directory error even though the file actually exists.
I am making use of Berkeley DB 4.7.25 for testing out these applications.
The four application names are as follows:
Writer 1
Writer 2
Reader 1
Reader 2
The application description is as follows:
‘*Writer 1*’ owns 8 environments and each environment having 123 Berkeley databases created using HASH access method. At any point of time, ‘*Writer 1*’ will be acting on 24 database files across 8 environments (3 database files per environment) for carrying out write operation. Where as reader process will be accessing all 123 database files / per environment (Total = 123 * 8 environments = 984 database files) for read activities. Similar configuration for Writer 2 as well – 8 separate environments and so on.
Writer 1, Reader 1 and Reader 2 processes share the environments created by Writer 1
Writer 2 and Reader 2 processes share the environments created by Writer 2
My DB_CONFIG file is configured as follows
set_cachesize 0 104857600 1   # 100 MB
set_lg_bsize 2097152                # 2 MB
set_data_dir ../../vol1/data/
set_lg_dir ../../vol31/logs/SUBID/
set_lk_max_locks 1500
set_lk_max_lockers 1500
set_lk_max_objects 1500
set_flags db_auto_commit
set_tx_max 200
mutex_set_increment 7500
Has anyone come across this problem before or is it something to do with the configuration?

Answer 1

Hi Michael,
I should have included about how we are making use of DB_TRUNCATE flag in my previous reply itself. Sorry for that.
From writers, DB truncation happens periodically. During truncate (DB handle is not associated with any environment handle i.e. stand-alone DB
) following parameters are passed to db->open function call:
DB->open(DB *db,                    
DB_TXN *txnid,          => NULL
const char *file,          => file name (absolute DB file path)
const char *database,  => NULL
DBTYPE type, => DB_HASH
u_int32_t flags, => DB_READ_UNCOMMITTED | DB_TRUNCATE | DB_CREATE | DB_THREAD
int mode); => 0
Also, DB_DUP flag is set.
As you have rightly pointed out, `No such file or directory` is occuring during truncation.
While a database is being truncated it will not be found by others trying to open it. We simulated this by stopping the writer process (responsible for truncation) and by opening & closing the databases repeatedly via readers. The reader process did not crash. When readers and writers were run simultaneously, we got `No such file or directory` error.Is there any solution to tackle this problem (because in our case writers and readers are run independently. So readers won't come to know about truncation from writers)?
Also, we are facing one more issue related to DB_TRUNCATE. Consider the following scenario:
<ul><li>Reader process holds reference of database (X) handle in environment 'Y' at time t1
</li>
<li>Writer process opens the database (X) in DB_TRUNCATE mode at time t2 (where t2 &gt; t1)</li>
<li>After truncation, writer process closes the database and joins the environment 'Y'</li>
<li>After this any writes to database X is not visible to the reader process</li>
<li>Once reader process closes the database X and re-joins the environment, all the records inserted from writer process are visible</li>
</ul>
Is it the right behavior? If yes, how to make any writes visible to a reader process without closing and re-opening the database in the above-mentioned scenario?
Also, when [db_set_errfile|http://www.oracle.com/technology/documentation/berkeley-db/db/api_c/db_set_errfile.html] was set, we did not get any additional information in the error file.
Thanks,
Magesh