WL 10.3 - Quartz's jobs stops after some time of correct executing

Question

Hi,
I have a webapp (backend only) that is deployed on WebLogic 10.3 AS. In this webapp 5 quartz's job are executed (cron trigger). Two of them run 3 times a day, the next two of them execute every 5 seconds and the last one starts every 20 seconds.
The problem is that after some time of correct execution (about 20 min.) one on the jobs hangs on, then after some time (about 10 min.) the second one hangs on and finally only one of job (it's always the "20" second job) executes correct for a long time, then also stops. Above, concerns the only the "5" and "20" seconds jobs. All of the jobs are stateful and Oracle DB is used for synchronisation.
In DB I see that "TRIGGER_STATE" in "TRIGGERS" tables has value of "BLOCKED" for the jobs that hang on. Whenever the job crashes I see the following entry in the WL's log:
####<2011-05-30 08:50:11 CEST> <Info> <RJVM> <server_name> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1306738211453> <BEA-000513> <Failure in heartbeat trigger for RJVM: -6468277378824263512S:127.0.1.1:[7001,7001,-1,-1,-1,-1,-1]:test_domain:test-server
java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl@559e - id: '-6468277378824263512S:127.0.1.1:[7001,7001,-1,-1,-1,-1,-1]:test_domain:test-server' connect time: 'Mon May 30 08:45:11 CEST 2011'' has already been shut down.
java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl@559e - id: '-6468277378824263512S:127.0.1.1:[7001,7001,-1,-1,-1,-1,-1]:test_domain:test-server' connect time: 'Mon May 30 08:45:11 CEST 2011'' has already been shut down
     at weblogic.rjvm.ConnectionManager.getOutputStream(ConnectionManager.java:1719)
     at weblogic.rjvm.ConnectionManager.createHeartbeatMsg(ConnectionManager.java:1662)
     at weblogic.rjvm.ConnectionManager.sendHeartbeatMsg(ConnectionManager.java:599)
     at weblogic.rjvm.RJVMImpl$HeartbeatChecker.timerExpired(RJVMImpl.java:1584)
     at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
     at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
     at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
     at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
>The quartz version I use is 1.8.4.
I use my own implementation of scheduler to start the jobs at the deploy time. Jobs are also always unscheduled when the app is undeployed. After undeploying all entries from quartz's tables are erased excepting the one in "SCHEDULER_STATE" table.
The webapp is tested standalone but it will be in the production environment it will be clustered (two nodes).
What can cause the problem? And how to solve it?
Below is my quartz.properties file as it's deployed with webapp:
#============================================================================
# Configure Main Scheduler Properties
#============================================================================
org.quartz.scheduler.instanceName = SimulatorClusteredScheduler
org.quartz.scheduler.instanceId = AUTO
#============================================================================
# Configure ThreadPool
#============================================================================
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 10
org.quartz.threadPool.threadPriority = 5
#============================================================================
# Configure JobStore
#============================================================================
org.quartz.jobStore.misfireThreshold = 60000
#org.quartz.jobStore.isClustered = true
#org.quartz.jobStore.clusterCheckinInterval = 20000
#org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreCMT
#org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.useProperties = false
org.quartz.jobStore.dataSource = myDS
org.quartz.jobStore.nonManagedTXDataSource = myDS
org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.jobStore.isClustered = true
org.quartz.jobStore.clusterCheckinInterval = 20000
#============================================================================
# Configure Datasources
#============================================================================
org.quartz.dataSource.myDS.jndiURL=${org.quartz.dataSource.myDS.jndiURL}
org.quartz.dataSource.myDS.jndiAlwaysLookup=true
org.quartz.dataSource.myDS.java.naming.factory.initial=weblogic.jndi.WLInitialContextFactory
org.quartz.dataSource.myDS.java.naming.provider.url=${org.quartz.dataSource.myDS.java.naming.provider.url}
org.quartz.dataSource.myDS.java.naming.security.principal=${org.quartz.dataSource.myDS.java.naming.security.principal}
org.quartz.dataSource.myDS.java.naming.security.credentials=${org.quartz.dataSource.myDS.java.naming.security.credentials}
#======================================================================
   # Configure Plugins
#======================================================================
#org.quartz.plugin.jobInitializer.overWriteExistingJobs = true
#org.quartz.plugin.shutdownhook.cleanShutdown = true Any help would be greatly appreciated.

Answer 1

Thanks for you response,
I have a entry, in web.xml, because of this this working at startup time. (connecting the quarts scheduler)
When server and application started every thing it's working but after some time(mostly after 30 minutes), without any change in server/application we are getting the above error.
We have entry in web.xml file like this.
<ejb-local-ref>
<ejb-ref-name>ejb/DXTrackingSession</ejb-ref-name>
<ejb-ref-type>Session</ejb-ref-type>
<local>com.sample.app.ejb.tracking.DXTrackingSession</local>
</ejb-local-ref>