Communication_failure_attempting_query_retry or BEA-001129

Question

I have a loop where I'm updating a column (happens to be a JGeometry). For each bean in a list I'm calling a mergeBean method as follows:
  public Object mergeBean(Object entityObject) throws Exception {
    final EntityManager em = getEm();
    try {
      final EntityTransaction et = em.getTransaction();
      try {
        em.clear();
        et.begin();
        entityObject = em.merge(entityObject);
        et.commit();
      } catch (Exception e) {
        _Logger.error("mergeBean failure ", e);
        if (et != null && et.isActive()) {
          et.rollback();
        throw e;
    } finally {
      closeEm();
    return object;
  }This works fine for about 30 or so beans and then I get three
<Feb 9, 2009 9:15:47 PM PST> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "precinctDS": Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
The Connection descriptor used by the client was:
merak.mizarllc.com:1521:merak
>followed by pages of
Feb 9, 2009 9:15:48 PM com.mizar.persistence.jpa.logging.MizarTopLinkLogger log
SEVERE: TopLink: communication_failure_attempting_query_retry
Feb 9, 2009 9:15:48 PM com.mizar.persistence.jpa.logging.MizarTopLinkLogger log
SEVERE: TopLink:
Feb 9, 2009 9:15:48 PM com.mizar.persistence.jpa.logging.MizarTopLinkLogger log
SEVERE: TopLink:
null
Feb 9, 2009 9:15:48 PM com.mizar.persistence.jpa.logging.MizarTopLinkLogger log
SEVERE: TopLink: communication_failure_attempting_query_retry
nullThese errors occur on some mergeBean requests and then some mergeBean calls work. Eventually I thrown an ADF error that we can ignore here but were
Caused by: java.lang.NullPointerException
     at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1219)
     at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1162)
     at org.eclipse.persistence.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:77)
     at org.eclipse.persistence.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:67)
     at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:83)
     at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:161)
     at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:230)
     at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:83)
     at org.eclipse.persistence.indirection.IndirectList.buildDelegate(IndirectList.java:213)
     at org.eclipse.persistence.indirection.IndirectList.getDelegate(IndirectList.java:379)
     at org.eclipse.persistence.indirection.IndirectList$1.<init>(IndirectList.java:526)
     at org.eclipse.persistence.indirection.IndirectList.listIterator(IndirectList.java:525)
     at org.eclipse.persistence.indirection.IndirectList.iterator(IndirectList.java:489)
     at com.mizar.addressing.tiger.state.Faces.getOuterEdges(Faces.java:450)
     at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at javax.el.BeanELResolver.getValue(BeanELResolver.java:261)
     ... 101 moreI tried commenting out the closeEm() so that I was reusing the same EntityManager and then I get this error.
SEVERE: mergeBean failure
javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool precinctDS is disabled, cannot allocate resources to applications..
Error Code: 0
     at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commitInternal(EntityTransactionImpl.java:102)
     at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:63)
     at com.mizar.persistence.jpa.facade.JPAFacade.mergeBean(JPAFacade.java:122)
     at com.mizar.trinidad.components.tiger.FacesTable.findFaces(FacesTable.java:55)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at com.sun.el.parser.AstValue.invoke(AstValue.java:157)
     at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:283)
     at org.apache.myfaces.trinidadinternal.taglib.util.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:53)
     at org.apache.myfaces.trinidad.component.UIXComponentBase.broadcastToMethodBinding(UIXComponentBase.java:1227)
     at org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:183)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:70)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:274)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:74)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:70)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:274)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:74)
     at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:458)
     at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:763)
     at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._invokeApplication(LifecycleImpl.java:640)
     at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:275)
     at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:175)
     at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
     at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
     at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
     at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
     at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
     at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
     at oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:85)
     at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:279)
     at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._invokeDoFilter(TrinidadFilterImpl.java:239)
     at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:196)
     at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:139)
     at org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
     at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
     at oracle.security.jps.wls.JpsWlsFilter.doFilter(JpsWlsFilter.java:102)
     at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:65)
     at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
     at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:149)
     at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
     at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
     at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
     at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496)
     at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
     at weblogic.security.service.SecurityManager.runAs(Unknown Source)
     at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
     at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
     at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
     at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
     at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.0.2 (Build 20081024)): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool precinctDS is disabled, cannot allocate resources to applications..
Error Code: 0
     at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:298)
     at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:138)
     at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:94)
     at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:164)
     at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:324)
     at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:264)
     at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:546)
     at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1372)
     at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:300)
     at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.rollbackTransaction(DatasourceAccessor.java:650)
     at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.rollbackTransaction(DatabaseAccessor.java:1432)
     at org.eclipse.persistence.internal.sessions.AbstractSession.basicRollbackTransaction(AbstractSession.java:473)
     at org.eclipse.persistence.sessions.server.ClientSession.basicRollbackTransaction(ClientSession.java:154)
     at org.eclipse.persistence.internal.sessions.AbstractSession.rollbackTransaction(AbstractSession.java:2762)
     at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.rollbackTransaction(UnitOfWorkImpl.java:4252)
     at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.rollbackTransaction(RepeatableWriteUnitOfWork.java:379)
     at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.rollbackTransaction(UnitOfWorkImpl.java:4273)
     at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabase(UnitOfWorkImpl.java:1290)
     at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitToDatabase(RepeatableWriteUnitOfWork.java:478)
     at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabaseWithChangeSet(UnitOfWorkImpl.java:1330)
     at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:159)
     at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1002)
     at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commitInternal(EntityTransactionImpl.java:84)
     ... 51 more
Caused by: weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool precinctDS is disabled, cannot allocate resources to applications..
     at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:250)
     at weblogic.jdbc.pool.Driver.connect(Driver.java:160)
     at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:642)
     at weblogic.jdbc.jts.Driver.connect(Driver.java:124)
     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:338)
     at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:133)
     ... 72 more
Source breakpoint occurred at line 134 of JPAFacade.java.While still in the same WebLogic session I can start the process up again and merge another 30 or so beans before it happens again. Sometimes I can call mergeBean as many as a hundred times between failures.
I took all the defaults when I created my pooled connector using the WebLogic console.
The following occurred while I was NOT calling closeEm() in my finally block.
I have 301 beans in my list. The first time I called the update it created 17 beans, the next time 124, the next time 155, and the final time 5 more during which I got this ConcurrentModificationException.
java.util.ConcurrentModificationException
     at java.util.IdentityHashMap$IdentityHashMapIterator.nextIndex(IdentityHashMap.java:715)
     at java.util.IdentityHashMap$KeyIterator.next(IdentityHashMap.java:804)
     at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.calculateChanges(UnitOfWorkImpl.java:547)
     at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:293)
     at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:384)
     at org.eclipse.persistence.internal.jpa.EJBQueryImpl.performPreQueryFlush(EJBQueryImpl.java:936)
     at org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:378)
     at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getResultList(EJBQueryImpl.java:517)
     at com.mizar.trinidad.components.tiger.FacesTable.findFaces(FacesTable.java:45)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at com.sun.el.parser.AstValue.invoke(AstValue.java:157)
     at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:283)
     at org.apache.myfaces.trinidadinternal.taglib.util.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:53)
     at org.apache.myfaces.trinidad.component.UIXComponentBase.broadcastToMethodBinding(UIXComponentBase.java:1227)
     at org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:183)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:70)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:274)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:74)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:70)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:274)
     at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:74)
     at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:458)
     at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:763)
     at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._invokeApplication(LifecycleImpl.java:640)
     at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:275)
     at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:175)
     at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
     at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
     at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
     at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
     at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
     at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
     at oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:85)
     at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:279)
     at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._invokeDoFilter(TrinidadFilterImpl.java:239)
     at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:196)
     at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:139)
     at org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
     at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
     at oracle.security.jps.wls.JpsWlsFilter.doFilter(JpsWlsFilter.java:102)
     at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:65)
     at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
     at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:149)
     at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
     at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
     at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
     at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496)
     at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
     at weblogic.security.service.SecurityManager.runAs(Unknown Source)
     at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
     at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
     at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
     at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
     at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)The specific number of beans that get merged changes with every restart of the application, however, some always get merged but never the entire 301. I have almost 600,000 of these beans to modify so I have to figure out why the pooled connectors are failing.
Any ideas?
Thanks, Mark

Answer 1

Yesterday we were able to narrow down the problem to the JGeometry properties. The problem occurs when a large number of queries are issued in a short period of time and the fetches contain JGeometry objects. In our case each FACES row had two EDGES lists each of which had two FACES (left and right). Navigating the EDGES of a FACES row created a lot of requests.
We were able to replicate the problem using EntityManager Query and re-executed the query inside the catch block. It would usually work on the second or third attempt.
The solution was to not use pooled connectors OR to re-write our code to not use the Eclipselink spatial extensions. We're going to not use pooled connectors for the immediate future while we're determining a time frame for Oracle to get WebLogic pooled connectors working for JGeometry.
Thanks, Mark