Berkeley DB

Question

Dear Ron,
Hi,
I have two databases that are created using Berkeley
DB, I need to merge these two databases into one
database. could you please tell me what is the fastest
way to do so?
currently, I am using the following method :
open the first database, search each key in the first
database in the second database, merge the data of
both keys and rewrite the data on the second database
but the speed is slow... I read about Cursor (Join
Union , Intersection ..) but I am not sure if those
operation are the best to get the best speed of
merging???
thank you for your help.
Best Regards,
Ahmad

Answer 1

Hi Alex,
Thank you for your quick response, I will adjust the database configuration (the pge size and the cache size) and I will inform you about the result as soon as possible.
now I have another problem :
I run the following code and I got the error shown after the end of this code:
package dbmerging;
import java.io.FileNotFoundException;
import com.sleepycat.db.LockMode;
import com.sleepycat.db.OperationStatus;
import com.sleepycat.db.Database;
import com.sleepycat.db.DatabaseConfig;
import com.sleepycat.db.DatabaseType;
import com.sleepycat.db.DatabaseException;
import com.sleepycat.db.Environment;
import com.sleepycat.db.EnvironmentConfig;
import com.sleepycat.db.DatabaseEntry;
import com.sleepycat.db.Cursor;
import com.sleepycat.db.CursorConfig;
import com.sleepycat.db.DatabaseException;
import com.sleepycat.db.MessageHandler;
import java.io.File;
import java.io.*;
import java.net.*;
import com.sleepycat.db.*;
public class Merging2DB {
public Merging2DB() {
class StringEntry extends DatabaseEntry {
StringEntry() {
StringEntry(String value) {
setString(value);
void setString(String value) {
byte[] data = value.getBytes();
setData(data);
setSize(data.length);
String getString() {
return new String(getData(), getOffset(), getSize());
String moddata="";
long sttime=0;
long entime=0;
long eltime =0 ;
/// try {
Cursor cursor =null;
Environment myDbE = null;
Environment myDbE1= null;
Database dbin = null;
Database dbout=null;
File envHome= new File("/database2006/index1");
File envHome1= new File("/database2006/index2");
String dbFileName = new String ("mydb.db");
try { // 1
// Open the environment. Create it if it does not already exist.
EnvironmentConfig envConfig = new EnvironmentConfig();
envConfig.setAllowCreate(true);
//envConfig.setCacheSize(131072);
// envConfig.setCacheSize(1048576);
envConfig.setCacheSize(2097152);
envConfig.setInitializeCache(true);
// Open the database. Create it if it does not already exist.
DatabaseConfig dbConfig = new DatabaseConfig();
dbConfig.setAllowCreate(true);
// dbConfig.setPageSize(32768);
dbConfig.setPageSize(8192);
dbConfig.setType(DatabaseType.BTREE);
myDbE = new Environment(envHome, envConfig);
myDbE1= new Environment(envHome1, envConfig);
// dbConfig.setSortedDuplicates(false);
dbin = myDbE.openDatabase(null, dbFileName,null, dbConfig);
dbout= myDbE1.openDatabase(null, dbFileName,null, dbConfig);
}//end try
catch(Exception e){System.out.println(e);}
sttime = System.currentTimeMillis();
StringEntry foundKey = new StringEntry();
StringEntry foundData = new StringEntry();
try {
cursor = dbout.openCursor(null,null);
while (cursor.getNext(foundKey, foundData, LockMode.DEFAULT) ==
OperationStatus.SUCCESS) {
String theKey = foundKey.getString();
String theData = foundData.getString();
try {
StringEntry searchdata = new StringEntry();
if (dbin.get(null, foundKey, searchdata,
LockMode.DEFAULT) ==
OperationStatus.SUCCESS) {
String retData = searchdata.getString();
// System.out.println("ret data= "+ retData);
moddata = retData +theData;
else moddata = theData;
try{
StringEntry updata = new StringEntry(moddata);
dbin.put(null,foundKey,updata);
}catch(DatabaseException dbe){System.out.println(dbe);}
catch (Exception e1) {System.out.println(e1);}
catch(DatabaseException dbe) {System.out.println(dbe);}
entime = System.currentTimeMillis();
eltime = entime - sttime;
System.out.println("\n eltime = " + eltime);
String etime = "";
etime = String.valueOf(eltime);
String wrp = "d://database2006//TimeMerging.txt";
try{
BufferedWriter br = new BufferedWriter(new FileWriter(wrp, true));
br.write("time of Merging = " + etime);
br.newLine();
br.close();
}catch(IOException e) { System.out.println(e);}
try {
if(cursor !=null) cursor.close();
if(myDbE !=null) myDbE.close();
if(myDbE1 !=null) myDbE1.close();
if(dbin !=null) dbin.close();
if(dbout !=null) dbout.close();
}catch(DatabaseException dbex){}
System.out.println("\n End of Merging.....");
public static void main(String[] args) {
Merging2DB merging2DB1 = new Merging2DB();
The error is :
java.lang.NullPointerException
     at java.lang.String.checkBounds(String.java:286)
     at java.lang.String.<init>(String.java:370)
     at dbmerging.Merging2DB$1$StringEntry.getString(Merging2DB.java:42)
     at dbmerging.Merging2DB.<init>(Merging2DB.java:90)
     at dbmerging.Merging2DB.main(Merging2DB.java:142)
Also when I defined a cursor to iterate over the same database above (index1)
the following error occurs :
"Database handles still open at environment close
Open database handle: mydb.db"
"An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : EXCEPTION_ACCESS_VIOLATION (0xc0000005) occurred at PC=0x2DCF723
Function=_db_add_recovery+0x442F
Library=D:\db-4.4.20.tar\db-4.4.20\build_win32\Debug\libdb44d.dll
Current Java thread:
     at com.sleepycat.db.internal.db_javaJNI.Dbc_close0(Native Method)
     at com.sleepycat.db.internal.Dbc.close0(Dbc.java:43)
     at com.sleepycat.db.internal.Dbc.close(Dbc.java:37)
     - locked <0x10055720> (a com.sleepycat.db.internal.Dbc)
     at com.sleepycat.db.Cursor.close(Cursor.java:36)
     - locked <0x100532e0> (a com.sleepycat.db.Cursor)
     at dbmerging.CursorExample.<init>(CursorExample.java:83)
     at dbmerging.CursorExample.main(CursorExample.java:94)
Dynamic libraries:
0x00400000 - 0x00407000      C:\jbuilderx\jdk1.4\bin\javaw.exe
0x7C900000 - 0x7C9B0000      C:\WINDOWS\system32\ntdll.dll
0x7C800000 - 0x7C8F4000      C:\WINDOWS\system32\kernel32.dll
0x77DD0000 - 0x77E6B000      C:\WINDOWS\system32\ADVAPI32.dll
0x77E70000 - 0x77F01000      C:\WINDOWS\system32\RPCRT4.dll
0x77D40000 - 0x77DD0000      C:\WINDOWS\system32\USER32.dll
0x77F10000 - 0x77F56000      C:\WINDOWS\system32\GDI32.dll
0x77C10000 - 0x77C68000      C:\WINDOWS\system32\MSVCRT.dll
0x629C0000 - 0x629C9000      C:\WINDOWS\system32\LPK.DLL
0x74D90000 - 0x74DFB000      C:\WINDOWS\system32\USP10.dll
0x08000000 - 0x08136000      C:\jbuilderx\jdk1.4\jre\bin\client\jvm.dll
0x76B40000 - 0x76B6D000      C:\WINDOWS\system32\WINMM.dll
0x10000000 - 0x10007000      C:\jbuilderx\jdk1.4\jre\bin\hpi.dll
0x00940000 - 0x0094E000      C:\jbuilderx\jdk1.4\jre\bin\verify.dll
0x00950000 - 0x00968000      C:\jbuilderx\jdk1.4\jre\bin\java.dll
0x00970000 - 0x0097D000      C:\jbuilderx\jdk1.4\jre\bin\zip.dll
0x02D60000 - 0x02D7F000      D:\db-4.4.20.tar\db-4.4.20\build_win32\Debug\libdb_java44d.dll
0x02D80000 - 0x02EA1000      D:\db-4.4.20.tar\db-4.4.20\build_win32\Debug\libdb44d.dll
0x02EB0000 - 0x02F10000      C:\WINDOWS\system32\MSVCRTD.dll
0x02F10000 - 0x02F8E000      C:\WINDOWS\system32\MSVCP60D.dll
0x76C90000 - 0x76CB8000      C:\WINDOWS\system32\imagehlp.dll
0x59A60000 - 0x59B01000      C:\WINDOWS\system32\DBGHELP.dll
0x77C00000 - 0x77C08000      C:\WINDOWS\system32\VERSION.dll
0x76BF0000 - 0x76BFB000      C:\WINDOWS\system32\PSAPI.DLL
Heap at VM Abort:
Heap
def new generation total 576K, used 358K [0x10010000, 0x100b0000, 0x104f0000)
eden space 512K, 57% used [0x10010000, 0x100598c0, 0x10090000)
from space 64K, 100% used [0x100a0000, 0x100b0000, 0x100b0000)
to space 64K, 0% used [0x10090000, 0x10090000, 0x100a0000)
tenured generation total 1408K, used 42K [0x104f0000, 0x10650000, 0x14010000)
the space 1408K, 3% used [0x104f0000, 0x104fa9a8, 0x104faa00, 0x10650000)
compacting perm gen total 4096K, used 1278K [0x14010000, 0x14410000, 0x18010000)
the space 4096K, 31% used [0x14010000, 0x1414f848, 0x1414fa00, 0x14410000)
Local Time = Sat Aug 05 15:53:58 2006
Elapsed Time = 0
# The exception above was detected in native code outside the VM
# Java VM: Java HotSpot(TM) Client VM (1.4.2_01-b06 mixed mode)
# An error report file has been saved as hs_err_pid3144.log.
# Please refer to the file for further information.
Exception in thread "main"
I checked for the existance of a key stored in the database which is ="" or null, and I found one key="". I do not know if this cause the abovbe problem or not?
Best regards,
Ahmad.
Message was edited by:
user525205