Multiple threads block each other, but its not a deadlock!

Question

I have recently upgraded to Berkeley 2.4.16/DB 4.6.21 on Linux 2.6.9-68.9.ELsmp #1 SMP 2008 x86_64 GNU/Linux
Right away I noticed that when I run two threads, one doing a putDocument, another doing a query, they get stuck, as if they were deadlocked, however running db_deadlock does not change the quagmire.
Here is the simplest class I could come up with that produces the problem I have been dealing with:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import com.sleepycat.db.DatabaseException;
import com.sleepycat.db.Environment;
import com.sleepycat.db.EnvironmentConfig;
import com.sleepycat.db.ReplicationManagerAckPolicy;
import com.sleepycat.db.ReplicationTimeoutType;
import com.sleepycat.dbxml.XmlContainer;
import com.sleepycat.dbxml.XmlContainerConfig;
import com.sleepycat.dbxml.XmlDocumentConfig;
import com.sleepycat.dbxml.XmlManager;
import com.sleepycat.dbxml.XmlManagerConfig;
import com.sleepycat.dbxml.XmlQueryContext;
import com.sleepycat.dbxml.XmlResults;
import com.sleepycat.dbxml.XmlUpdateContext;
import com.sleepycat.dbxml.XmlValue;
public class TestEnvironment
    private XmlManager i_xmlManager = null;
    private XmlContainer i_container = null;
    private XmlContainerConfig i_xmlContainerConfig = null;
    private XmlDocumentConfig i_docCfg = null;
    public TestEnvironment(File dataDir, File dbErr, File dbOut)
    throws DatabaseException, FileNotFoundException
        final EnvironmentConfig cfg = new EnvironmentConfig();
        cfg.setErrorStream(new FileOutputStream(dbErr));
        cfg.setMessageStream(new FileOutputStream(dbOut));
        cfg.setAllowCreate(true);
        cfg.setInitializeLocking(true);
        cfg.setInitializeLogging(true);
        cfg.setInitializeCache(true);
        cfg.setTransactional(true);
        cfg.setRunRecovery(false);
        cfg.setTxnNoSync(true);
        cfg.setTxnNotDurable(false);
        cfg.setTxnTimeout(60000000L);
        cfg.setCacheSize(1073741824L);
        cfg.setThreaded(true);
        cfg.setInitializeReplication(false);
        cfg.setReplicationLimit(1048576L);
        cfg.setVerboseReplication(true);
        cfg.setReplicationManagerAckPolicy(ReplicationManagerAckPolicy.NONE);
        cfg.setMaxLockers(100000);
        cfg.setMaxLockObjects(100000);
        cfg.setMaxLocks(100000);
        cfg.setLockDown(false);
        cfg.setSystemMemory(false);
        cfg.setInitializeCDB(false);
        final Environment env = new Environment(dataDir, cfg);
        env.setReplicationTimeout(ReplicationTimeoutType.ACK_TIMEOUT,  100000);
        env.setReplicationTimeout(ReplicationTimeoutType.CONNECTION_RETRY, 100000);
        env.setReplicationTimeout(ReplicationTimeoutType.ELECTION_RETRY, 100000);
        env.setReplicationTimeout(ReplicationTimeoutType.ELECTION_TIMEOUT, 90000);
        final XmlManagerConfig mgrCfg = new XmlManagerConfig();
        mgrCfg.setAdoptEnvironment(true);
        mgrCfg.setAllowAutoOpen(true);
        mgrCfg.setAllowExternalAccess(false);
        XmlManager.setLogCategory(XmlManager.CATEGORY_ALL, true);
        XmlManager.setLogLevel(XmlManager.LEVEL_ALL, true);
        i_xmlManager = new XmlManager(env, mgrCfg);
        i_xmlManager.setDefaultContainerType(XmlContainer.NodeContainer);
        i_xmlContainerConfig = new XmlContainerConfig();
        i_xmlContainerConfig.setAllowValidation(false);
        i_xmlContainerConfig.setIndexNodes(true);
        i_xmlContainerConfig.setNodeContainer(true);
        i_xmlContainerConfig.setTransactional(true);  
        if (i_xmlManager.existsContainer("container.dbxml") != 0)
            i_container = i_xmlManager.openContainer("container.dbxml", i_xmlContainerConfig);
        else
            i_container = i_xmlManager.createContainer("container.dbxml", i_xmlContainerConfig);
        i_docCfg = new XmlDocumentConfig();
        i_docCfg.setGenerateName(true);
        final TestEnvironment thisRef = this;
        Runtime.getRuntime().addShutdownHook(new Thread()
            public void run()
                try
                    thisRef.close();
                    System.out.println("Shutting down the TestEnvironment.");
                catch (Exception e)
                    e.printStackTrace();
    public void close() throws DatabaseException
        if (i_container != null)
            i_container.close();
            i_container = null;
        if (i_xmlManager != null)
            i_xmlManager.close();
            i_xmlManager = null;
    public void insert(String doc) throws DatabaseException
        System.out.println('[' + Thread.currentThread().getName() +
        "] insert received document to be inserted");
        final long beforeT = System.currentTimeMillis();
        final XmlUpdateContext ctxt = i_xmlManager.createUpdateContext();
        i_container.putDocument(null, doc, ctxt, i_docCfg);
        final long afterT = System.currentTimeMillis();
        System.out.println('[' + Thread.currentThread().getName() +
            "] insert  took " + (afterT - beforeT) + " ms. ");
    public String[] query(String xquery) throws DatabaseException
        System.out.println('[' + Thread.currentThread().getName() +
            "] query \"" + xquery + "\" received.");
        String[] retVal = {};
        final long beforeT = System.currentTimeMillis();
        XmlQueryContext qctxt = null;
        XmlResults rs = null;
        XmlValue nextValue = null;
        try
            qctxt = i_xmlManager.createQueryContext();
            qctxt.setQueryTimeoutSeconds(10);
            rs = i_xmlManager.query(xquery, qctxt);
            if (rs != null)
                retVal = new String[rs.size()];
                for  (int i = 0; i < retVal.length && rs.hasNext(); i++)
                    nextValue = rs.next();
                    retVal[i] = nextValue.asString();
                    nextValue.delete();
                    nextValue = null;
        finally
            if (nextValue != null)
                nextValue.delete();
            if (qctxt != null)
                qctxt.delete();
            if (rs != null)
                rs.delete();
        final long afterT = System.currentTimeMillis();
        System.out.println('[' + Thread.currentThread().getName() +
            "] query \"" + xquery + "\" took " + (afterT - beforeT) + " ms. ");
        return retVal;
}If I call the insert and the query methods in parallel from two different threads, they will both be stuck, the former in the method com.sleepycat.dbxml.dbxml_javaJNI.XmlContainer_putDocument__SWIG_0(Native Method) and the latter in the method com.sleepycat.dbxml.dbxml_javaJNI.XmlManager_query__SWIG_2(Native Method). I would really appreciate help with this issue, I've looked through all compilation flags for db and dbxml, and all runtime configuration parameters, and I can not figure out why this is not working. Thank you!
Edited by: gmfeinberg on Mar 26, 2009 10:41 AM for formatting

Answer 1

Upon your suggestion, I added explicit transactions to my code, and now it looks like this:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import com.sleepycat.db.DatabaseException;
import com.sleepycat.db.Environment;
import com.sleepycat.db.EnvironmentConfig;
import com.sleepycat.db.LockDetectMode;
import com.sleepycat.dbxml.XmlContainer;
import com.sleepycat.dbxml.XmlContainerConfig;
import com.sleepycat.dbxml.XmlDocumentConfig;
import com.sleepycat.dbxml.XmlException;
import com.sleepycat.dbxml.XmlManager;
import com.sleepycat.dbxml.XmlManagerConfig;
import com.sleepycat.dbxml.XmlQueryContext;
import com.sleepycat.dbxml.XmlResults;
import com.sleepycat.dbxml.XmlTransaction;
import com.sleepycat.dbxml.XmlValue;
public class TestEnvironment
    private static final int DEADLOCK_DETECTOR_INTERVAL = 5000;
    private XmlManager i_xmlManager = null;
    private XmlContainer i_container = null;
    private XmlContainerConfig i_xmlContainerConfig = null;
    private XmlDocumentConfig i_docCfg = null;
    private boolean i_shuttingDown = false;
    public TestEnvironment(File dataDir, File dbErr, File dbOut)
    throws XmlException, DatabaseException, FileNotFoundException
        final EnvironmentConfig cfg = new EnvironmentConfig();
        cfg.setErrorStream(new FileOutputStream(dbErr));
        cfg.setMessageStream(new FileOutputStream(dbOut));
        cfg.setAllowCreate(true);
        cfg.setInitializeLocking(true);
        cfg.setInitializeLogging(true);
        cfg.setInitializeCache(true);
        cfg.setTransactional(true);
        cfg.setRunRecovery(false);
        cfg.setTxnNoSync(true);
        cfg.setTxnNotDurable(false);
        //cfg.setTxnTimeout(500000L);
        cfg.setCacheSize(1073741824L);
        cfg.setThreaded(true);
        cfg.setInitializeReplication(false);
        cfg.setMaxLockers(100000);
        cfg.setMaxLockObjects(100000);
        cfg.setMaxLocks(100000);
        cfg.setLockDown(false);
        cfg.setSystemMemory(false);
        cfg.setInitializeCDB(false);
        final Environment env = new Environment(dataDir, cfg);
        final XmlManagerConfig mgrCfg = new XmlManagerConfig();
        mgrCfg.setAdoptEnvironment(true);
        mgrCfg.setAllowAutoOpen(true);
        mgrCfg.setAllowExternalAccess(false);
        XmlManager.setLogCategory(XmlManager.CATEGORY_ALL, true);
        XmlManager.setLogLevel(XmlManager.LEVEL_ALL, true);
        i_xmlManager = new XmlManager(env, mgrCfg);
        i_xmlManager.setDefaultContainerType(XmlContainer.NodeContainer);
        i_xmlContainerConfig = new XmlContainerConfig();
        i_xmlContainerConfig.setAllowValidation(false);
        i_xmlContainerConfig.setIndexNodes(true);
        i_xmlContainerConfig.setNodeContainer(true);
        i_xmlContainerConfig.setTransactional(true);  
        i_xmlContainerConfig.setStatisticsEnabled(false);
        if (i_xmlManager.existsContainer("container.dbxml") != 0)
            i_container = i_xmlManager.openContainer("container.dbxml", i_xmlContainerConfig);
        else
            i_container = i_xmlManager.createContainer("container.dbxml", i_xmlContainerConfig);
        i_docCfg = new XmlDocumentConfig();
        i_docCfg.setGenerateName(true);
        final TestEnvironment thisRef = this;
        Runtime.getRuntime().addShutdownHook(new Thread()
            public void run()
                try
                    thisRef.close();
                    System.out.println("Shutting down the TestEnvironment.");
                catch (Exception e)
                    e.printStackTrace();
        final Thread deadLockDetector = new Thread("deadLockDetector") {
            @Override
            public void run()
                while (!i_shuttingDown)
                    try
                        i_xmlManager.getEnvironment().detectDeadlocks(LockDetectMode.YOUNGEST);
                        System.out.println('[' + Thread.currentThread().getName() +
                            "] ran deadlock detector.");
                        Thread.sleep(DEADLOCK_DETECTOR_INTERVAL);
                    catch (XmlException e)
                        e.printStackTrace();
                    catch (DatabaseException e)
                        e.printStackTrace();
                    catch (InterruptedException e)
                        e.printStackTrace();
        deadLockDetector.start();
    public void close() throws XmlException
        i_shuttingDown = true;
        if (i_container != null)
            i_container.close();
            i_container = null;
        if (i_xmlManager != null)
            i_xmlManager.close();
            i_xmlManager = null;
    public void insert(String doc) throws XmlException
        System.out.println('[' + Thread.currentThread().getName() +
            "] insert received document to be inserted");
        final long beforeT = System.currentTimeMillis();
        final XmlTransaction txn = i_xmlManager.createTransaction();
        try
            i_container.putDocument(txn, null, doc, i_docCfg);
            txn.commit();
        catch (XmlException e)
            txn.abort();
            throw e;
        finally
            txn.delete();
        final long afterT = System.currentTimeMillis();
        System.out.println('[' + Thread.currentThread().getName() +
            "] insert  took " + (afterT - beforeT) + " ms. ");
    public String[] query(String xquery) throws XmlException
        System.out.println('[' + Thread.currentThread().getName() +
            "] query \"" + xquery + "\" received.");
        String[] retVal = {};
        final long beforeT = System.currentTimeMillis();
        XmlQueryContext qctxt = null;
        XmlResults rs = null;
        XmlValue nextValue = null;
        final XmlTransaction txn = i_xmlManager.createTransaction();
        try
            qctxt = i_xmlManager.createQueryContext();
            qctxt.setQueryTimeoutSeconds(10);
            rs = i_xmlManager.query(txn, xquery, qctxt);
            if (rs != null)
                retVal = new String[rs.size()];
                for  (int i = 0; i < retVal.length && rs.hasNext(); i++)
                    nextValue = rs.next();
                    retVal[i] = nextValue.asString();
                    nextValue.delete();
                    nextValue = null;
            txn.commit();
        catch (XmlException e)
            txn.abort();
            throw e;
        finally
            txn.delete();
            if (nextValue != null)
                nextValue.delete();
            if (qctxt != null)
                qctxt.delete();
            if (rs != null)
                rs.delete();
        final long afterT = System.currentTimeMillis();
        System.out.println('[' + Thread.currentThread().getName() +
            "] query \"" + xquery + "\" took " + (afterT - beforeT) + " ms. ");
        return retVal;
}For the purpose of brevity I omitted the main method, but it merely runs two parallel threads -- one inserting a collection of different XML documents, the other -- querying. Each thread runs a semi-tight loop with a 10 millisecond sleep after each iteration. The documents being inserted are all fairly similar, and well-formed. Here is what happens: if I do not set a maximum transaction timeout, every SINGLE concurrent pair of inserts/queries results in a deadlock, that is only broken when the deadlock detector thread runs every five seconds. Our application does require massive inserts/quieries run in parallel sometimes. If I do set a maximum transaction timeout that expires before the deadlock detector thread runs, I get the following exception:
com.sleepycat.dbxml.XmlException: Error: DB_LOCK_NOTGRANTED: Lock not granted, errcode = DATABASE_ERROR
        at com.sleepycat.dbxml.dbxml_javaJNI.XmlContainer_putDocument__SWIG_3(Native Method)
        at com.sleepycat.dbxml.XmlContainer.putDocument(XmlContainer.java:736)
        at com.sleepycat.dbxml.XmlContainer.putDocument(XmlContainer.java:232)
        at com.sleepycat.dbxml.XmlContainer.putDocument(XmlContainer.java:218)
        at TestEnvironment.insert(TestEnvironment.java:178)
        at TestEnvironment$3.run(TestEnvironment.java:327)which does not seem to be a deadlock.
Now I understand that deadlocks are a fact of life, but I thought they were statistically rare, and in this scenario every single insert/query pair results in a deadlock. The same code works without any deadlocks in the Berkeley DB XML 2.3/Berkeley DB 4.5 combination
Could there be a patch or a bugfix that I missed? This seems very very suspicious. I also tried the same code on both Linux and Windows with the same result
Thank you,
Alexander.