Terminating threads
Hi,
I have developed a simple program to test executor service and executors for thread pool creation and management. The program executes fine until I call the shudown() method. The Executor is shutdown but the threads still remain alive (verified by calling Shutdown() -> isShutdown() -> isTerminated())
I have supplied the code along with this message:
import java.util.concurrent.*;
public class PerfTrack {
public static void main(String args[]){
final int CAPACITY = 5;
final ExecutorService threadPool;
Tasks hasTasks = new Tasks();
ArrayBlockingQueue taskQueue = new ArrayBlockingQueue(CAPACITY, true);
threadPool = Executors.newFixedThreadPool(10);
threadPool.execute(new Scheduler(taskQueue));
while(hasTasks.getTasks())
threadPool.execute(new Handler(taskQueue, hasTasks));
threadPool.shutdown();
System.out.println("hello");
System.out.println(threadPool.isShutdown());
System.out.println(threadPool.isTerminated());
class Tasks{
private boolean hasTasks;
public Tasks(){
hasTasks = true;
public synchronized boolean getTasks(){
return hasTasks;
public synchronized void setTasks(){
hasTasks = false;
class Scheduler implements Runnable{
private ArrayBlockingQueue taskQueue;
public Scheduler(ArrayBlockingQueue taskQueue){
this.taskQueue = taskQueue;
private void createTasks(){
for(int i = 1; i <= 20; i++){
try{
taskQueue.put(new Integer(i));
if(i == 20){
taskQueue.put(new Integer(0));
catch(InterruptedException intEx){
// Do nothing
public void run(){
createTasks();
class Handler implements Runnable{
private ArrayBlockingQueue taskQueue;
private Integer queueItem;
volatile Tasks hasTasks;
public Handler(ArrayBlockingQueue taskQueue, Tasks hasTasks){
this.taskQueue = taskQueue;
this.hasTasks = hasTasks;
private synchronized void getItems(){
try{
queueItem = (Integer)taskQueue.take();
if(queueItem.intValue() == 0){
hasTasks.setTasks();
else
System.out.println("The number is:\t" + queueItem.intValue());
catch(InterruptedException intEx){
public void run(){
getItems();
}Your help is greatily appreciated,
Thanks
Java does not provide any mechanism for safely forcing a thread to stop what it is doing.[1] Instead, it provides interruption, a cooperative mechanism that lets one thread ask another to stop what it is doing.
The deprecated Thread.stop and suspend methods were an attempt to provide such a mechanism, but were quickly realized to be seriously flawed and should be avoided. See http://java.sun.com/j2se/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html for an explanation of the problems with these methods.
public class PrimeGenerator implements Runnable {
private final List<BigInteger> primes
= new ArrayList<BigInteger>();
private volatile boolean cancelled;
public void run() {
BigInteger p = BigInteger.ONE;
while (!cancelled ) {
p = p.nextProbablePrime();
synchronized (this) {
primes.add(p);
public void cancel() { cancelled = true; }
public synchronized List<BigInteger> get() {
return new ArrayList<BigInteger>(primes);
}♠
Similar Messages
-
VPN error "Debugger terminated thread"
All of a sudden I am getting "Debugger terminated thread" while connecting to my office network. First it prompts for a password then
refuses the correct one a few times. This is very urgent.
ThanksHi Gregpakes,
There seems to be strange.
If they are configured with the similar settings, they should all work.
Besides, if we have configured any 3rd-party VPN software, there might be some settings that we may take care with, so better confirm with the manufacturer side.
Another thing that may helps is try to take use of Network Monitor and the process monitor to figure out what is going on with the two VPNs.
Process Monitor v3.1
Microsoft Network Monitor 3.4
Regards
Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact [email protected] -
Terminating threads cleanly after a period of time
Hey,
Im working on an assignment
"Consider the CarPark control problem from the lecture notes?(available online).
You are required to extend this example to model a car park having 400 spaces
and 3 entrances and 3 exits. This means 3 threads must be created to simulate
the arrival of cars at each entrance and a further 3 threads must be created to
simulate the departure of cars at each exit. Each thread should sleep for a period
of roughly 0.5s between each attempt to perform an arrival or a departure. The
program should terminate all threads (cleanly) after 180s and the current number
of spaces in the car park should be printed out, and the total number of cars that
have entered and left by each entrance and exit should also be printed out. The
aim of this is a rough check to determine if your code is correct."
So I have everything done and working but I am having difficulty with terminating the threads, it seems that thread.stop() doesnt work, and I think that thread.interrupt() should do it but Im not sure how?!
Any thoughts would be greatly appreciated
ThanksAccording to the API the correct way to do it now is setting the thread to null.
If you set your threads up correctly they will complete their processing and you can set them to null without interrupting them.
for instance.
in the run() method have a boolean value as an on/off switch.
while(on)
... do stuff
}It will continually run until on is set to false. It runs itself out and you can set the thread object to null and garbage collection handles the rest. -
Terminating thread at test completion
I have a sequence that kicks off a VI (contained in its own subsequence) in a separate thread. The tests that run after that talk to this separately-threaded VI. They report pass/fail and the test ends. However, until that separately-threaded VI stops, the sequence doesn't complete (and show the test results). Fortunately I display the VI at runtime and I have a STOP on the front panel (so I don't have to abort). However, I'd like a more graceful way to stop that thread. Thoughts?
You have to include interthread-communication. If the testing VI is running in the same application instance as the "serving VI", you can use queues to transfer data and commands. If they do not run in the same application instance, you can use protocols like TCP.
Terminating the TestStand Execution is no solution since the thread in TestStand, which called the service VI, is not active and therefor cannot be terminated until the VI is finished.
hope this helps,
Norbert
CEO: What exactly is stopping us from doing this?
Expert: Geometry
Marketing Manager: Just ignore it. -
Thread programming - variable might not have been initialized
Hi there,
Probably a fairly simple one here (quite new to java), i'm trying to do some thread programming but I get this error.
ThreadController.java:42: variable dataGeneratorThread might not have been initialized
dataGeneratorThread.activeCount();
The bit I dont understand is that I run
this line here
for(int i=0; i < JuliaFrame.NO_OF_DATA_GENERATORS; i++){
dataGenerators[i] = new DataGenerator(JuliaFrame.ImageWidth,JuliaFrame.ImageHeight,inputBuffer);
dataGeneratorThread = new WorkerThread(dataGenerators,juliaCalc,displayBuffer);
dataGeneratorThread.start();
but when I do
dataGeneratorThread.activeCount()
is balks.
I check that the thread is alive first but dont no how to tell the compiler this - obviously NO_OF_DATA_GENERATORS could be 0 in which case the compiler would have issues - I just wish it was more trusting ;)
Here is some of the code
runable class{
new Thread( new ThreadController(inputBuffer,displayBuffer,juliaCalc) ).start();
public class ThreadController implements Runnable {
DataBuffer inputBuffer;
DataBuffer displayBuffer;
JuliaCalc juliaCalc;
/** Creates a new instance of ThreadController */
public ThreadController(DataBuffer inputBuffer,DataBuffer displayBuffer,JuliaCalc juliaCalc) {
inputBuffer = inputBuffer;
displayBuffer = displayBuffer;
juliaCalc = juliaCalc;
public void run()
DataGenerator[] dataGenerators = new DataGenerator[JuliaFrame.NO_OF_DATA_GENERATORS];
WorkerThread dataGeneratorThread;
for(int i=0; i < JuliaFrame.NO_OF_DATA_GENERATORS; i++){
dataGenerators[i] = new DataGenerator(JuliaFrame.ImageWidth,JuliaFrame.ImageHeight,inputBuffer);
dataGeneratorThread = new WorkerThread(dataGenerators[i],juliaCalc,displayBuffer);
dataGeneratorThread.start();
dataGeneratorThread.activeCount();
int nNanoSecsBeforeCheck = 50;
while(true)
if(dataGeneratorThread.isAlive()){
if(dataGeneratorThread.activeCount() > inputBuffer.size())
public class DataGenerator implements Runnable {
int nImageWidth;
int nImageHeight;
DataBuffer inputBuffer;
public DataGenerator(int ImageWidth, int ImageHeight,DataBuffer inputBuffer) {
nImageWidth = ImageWidth;
nImageHeight = ImageHeight;
public void run()
// Non terminating thread - created datapakets and place them in the input buffer
// Get two radom number withoin a bound and find out bout default highest colou - 3rd value
while(true)
int nRandomXCord;
int nRandomYCord;
nRandomXCord = ((int)(Math.random() * nImageWidth));
nRandomYCord = ((int)(Math.random() * nImageHeight));
DataPacket dpDataPacket = new DataPacket(nRandomXCord, nRandomYCord, 233);
inputBuffer.put(dpDataPacket);Sorry about that, didnt realise.
Hi there,
Probably a fairly simple one here (quite new to java), i'm trying to do some thread programming but I get this error.
ThreadController.java:42: variable dataGeneratorThread might not have been initialized
The bit I dont understand is that I run
this line here
this line here
for(int i=0; i < JuliaFrame.NO_OF_DATA_GENERATORS; i++){
dataGenerators = new DataGenerator(JuliaFrame.ImageWidth,JuliaFrame.ImageHeight,inputBuffer);
dataGeneratorThread = new WorkerThread(dataGenerators,juliaCalc,displayBuffer);
dataGeneratorThread.start();
}but when I do
dataGeneratorThread.activeCount() is balks.
I check that the thread is alive first but dont no how to tell the compiler this - obviously NO_OF_DATA_GENERATORS could be 0 in which case the compiler would have issues - I just wish it was more trusting ;)
Here is some of the code
runable class{
new Thread( new ThreadController(inputBuffer,displayBuffer,juliaCalc) ).start();
public class ThreadController implements Runnable {
DataBuffer inputBuffer;
DataBuffer displayBuffer;
JuliaCalc juliaCalc;
/** Creates a new instance of ThreadController */
public ThreadController(DataBuffer inputBuffer,DataBuffer displayBuffer,JuliaCalc juliaCalc) {
inputBuffer = inputBuffer;
displayBuffer = displayBuffer;
juliaCalc = juliaCalc;
public void run()
DataGenerator[] dataGenerators = new DataGenerator[JuliaFrame.NO_OF_DATA_GENERATORS];
WorkerThread dataGeneratorThread;
for(int i=0; i < JuliaFrame.NO_OF_DATA_GENERATORS; i++){
dataGenerators = new DataGenerator(JuliaFrame.ImageWidth,JuliaFrame.ImageHeight,inputBuffer);
dataGeneratorThread = new WorkerThread(dataGenerators,juliaCalc,displayBuffer);
dataGeneratorThread.start();
dataGeneratorThread.activeCount();
int nNanoSecsBeforeCheck = 50;
while(true)
if(dataGeneratorThread.isAlive()){
if(dataGeneratorThread.activeCount() > inputBuffer.size())
public class DataGenerator implements Runnable {
int nImageWidth;
int nImageHeight;
DataBuffer inputBuffer;
public DataGenerator(int ImageWidth, int ImageHeight,DataBuffer inputBuffer) {
nImageWidth = ImageWidth;
nImageHeight = ImageHeight;
public void run()
// Non terminating thread - created datapakets and place them in the input buffer
// Get two radom number withoin a bound and find out bout default highest colou - 3rd value
while(true)
int nRandomXCord;
int nRandomYCord;
nRandomXCord = ((int)(Math.random() * nImageWidth));
nRandomYCord = ((int)(Math.random() * nImageHeight));
DataPacket dpDataPacket = new DataPacket(nRandomXCord, nRandomYCord, 233);
inputBuffer.put(dpDataPacket);
} -
Why are the threads start and terminate randomly?
Hi there,
I got the program below. I am wondering why are the threads start and terminate randomly? Everytime, I run the program, it produces different results.
I know that these four threads have got same normal priority (should be 5), and under windows there is something called timeslice. Then these four threads rotate using this timeslice. How do we know what exactly the timeslice is in seconds? If the timeslice is fix, then why the results are ramdom?
Thanks in advance!
* To change this template, choose Tools | Templates
* and open the template in the editor.
package mythreadone;
* @author Administrator
public class MyThreadOne implements Runnable {
String tName;
Thread t;
MyThreadOne(String threadName) {
tName = threadName;
t = new Thread(this, tName);
t.start();
public void run() {
try {
System.out.println("Thread: " + tName);
Thread.sleep(2000);
} catch (InterruptedException e) {
System.out.println("Exception: Thread "
+ tName + " interrupted");
System.out.println("Terminating thread: " + tName);
public static void main(String args[]) {
// Why are the threads start and terminate randomly?
new MyThreadOne("1");
new MyThreadOne("2");
new MyThreadOne("3");
new MyThreadOne("4");
try {
Thread.sleep(10000);
// Thread.sleep(2000);
} catch (InterruptedException e) {
System.out.println(
"Exception: Thread main interrupted.");
System.out.println(
"Terminating thread: main thread.");
1. Firstly, I set in the main function:
Thread.sleep(10000);
and I run the program it gives:
Thread: 1
Thread: 4
Thread: 2
Thread: 3
Terminating thread: 1
Terminating thread: 3
Terminating thread: 4
Terminating thread: 2
Terminating thread: main thread.
BUILD SUCCESSFUL (total time: 10 seconds)
Run it again, it gives:
Thread: 2
Thread: 4
Thread: 3
Thread: 1
Terminating thread: 2
Terminating thread: 1
Terminating thread: 3
Terminating thread: 4
Terminating thread: main thread.
BUILD SUCCESSFUL (total time: 10 seconds)
And my question was why it outputs like this? It suppose to be:
Thread: 1
Thread: 2
Thread: 3
Thread: 4
Terminating thread: 1
Terminating thread: 2
Terminating thread: 3
Terminating thread: 4
Terminating thread: main thread.
BUILD SUCCESSFUL (total time: 10 seconds)
Why these four threads start and finish randomly each time I run the program? I use Windows, suppose there is a timeslice (i.e. 1 second), these threads have the same priority. Then the threads should start and finish in turn one by one. Am I right?
2. My second question is:
When I change the codes in the 'main' function into:
Thread.sleep(10000); -> Thread.sleep(2000);
it gives me the results like:
Thread: 1
Thread: 4
Thread: 3
Thread: 2
Terminating thread: main thread.
Terminating thread: 1
Terminating thread: 4
Terminating thread: 3
Terminating thread: 2
BUILD SUCCESSFUL (total time: 2 seconds)
Run it again:
Thread: 1
Thread: 2
Thread: 3
Thread: 4
Terminating thread: 3
Terminating thread: main thread.
Terminating thread: 4
Terminating thread: 2
Terminating thread: 1
BUILD SUCCESSFUL (total time: 2 seconds)
I tried several times. The main thread always terminates before or after the first child thread finished.
My question is why it doesn't output something like:
Thread: 1
Thread: 2
Thread: 3
Thread: 4
Terminating thread: 3
Terminating thread: 4
Terminating thread: 2
Terminating thread: main thread.
Terminating thread: 1
BUILD SUCCESSFUL (total time: 2 seconds)
or
Thread: 1
Thread: 2
Thread: 3
Thread: 4
Terminating thread: 3
Terminating thread: 4
Terminating thread: 2
Terminating thread: 1
Terminating thread: main thread.
BUILD SUCCESSFUL (total time: 2 seconds)user13476736 wrote:
Yes, my machine has multi-core. Then you mean that if I got a one core machine the result should always be:
Thread: 1
Thread: 2
Thread: 3
Thread: 4
Terminating thread: 1
Terminating thread: 2
Terminating thread: 3
Terminating thread: 4
Terminating thread: main thread.
BUILD SUCCESSFUL (total time: 10 seconds)
???No.
>
How to explain my second quesiton then? Why the main thread always terminates before some of the child threads end? Thanks a lot. -
How to get exit status of multiple threads
Hello Everyone,
Please let me know how to solve the below problem.
I have one main thread. It creates 10 threads and joins them. Now i am interested in the exit status of each of the individual thread. Please let me know how to get the exit status of each thread.
Thanks in Advance.
Thanks & Regards,
VasuFrom the man page for thr_join:
int thr_join(thread_t thread, thread_t departed, void *status);
If a thr_join() call returns successfully with a non-null
status argument, the value passed to thr_exit(3THR) by the
terminating thread will be placed in the location referenced
by status.
To find the man page for any of the thr_xxx functions, run the command
man thr_xxx -
package com.bindhuw.saro.corpactions.control;
import java.util.Date;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.commons.logging.impl.Log4JLogger;
import org.apache.log4j.LogManager;
import com.bindhuw.saro.common.saroMultiProcess;
* @author
* @version 1.0
* @Description saro Application Main class for the Event Manager background process. The class runs the following 4 threads:
* CAInitialReconGenerator,CaStatementGenerator CaTaskGenerator & CATaskController
public class CAEventManager extends saroMultiProcess {
// Name of the process
public static final String PROCESS_NAME = "caEventManager";
* The instance of logger for logging
// private static Category logger;
private static Log logger;
* Constructor of the class.
* @param port Port number
* @param envId Environment id.
* @param processId Process id.
* @param termId Terminal id.
* @param procName Process name.
public CAEventManager(int port, int envId, int processId, String termId, String procName) {
super(port, envId, processId, termId, procName);
* Main fucntion to start the service.
* @param args Command line arguments.
static public void main(String[] args) {
final int NO_OF_PARAMS = 5;
// check for valid number of parameters
if (args.length < NO_OF_PARAMS) {
System.err.println("Invalid number of command line parameters: " + args.length);
System.err.println("Expected number of command line parameters: " + NO_OF_PARAMS);
System.exit(1);
logger = LogFactory.getLog(PROCESS_NAME);
// print some messages
String msgPrefix = "[SYSTEM][" + PROCESS_NAME + "]";
logger.fatal(msgPrefix + "Process starting...");
Date now = new Date();
logger.info("************************************************************");
logger.info("* Process : " + PROCESS_NAME);
logger.info("* Time Stamp: " + now);
logger.info("************************************************************");
try {
System.setProperty("CONFIG_BASE", args[4]);
// load properties
loadPropertiesFor(PROCESS_NAME);
logger.info("Loaded properties from file: " + getPropertiesFile());
int port = Integer.parseInt(args[0]);
int envId = Integer.parseInt(args[1]);
int processId = Integer.parseInt(args[2]);
CAEventManager eventMgr = new CAEventManager(port, envId, processId, args[3], PROCESS_NAME);
eventMgr.getChildProcesses().add(new CAInitialReconGenerator(eventMgr));
eventMgr.getChildProcesses().add(new CAStatementGenerator(eventMgr));
eventMgr.getChildProcesses().add(new CATaskGenerator(eventMgr));
eventMgr.getChildProcesses().add(new CATaskController(eventMgr));
eventMgr.getChildProcesses().add(new CADirectoryWatcher(eventMgr));
eventMgr.startServer();
} catch (Throwable t) {
logger.error(msgPrefix + "Server startup exception: " + t.getMessage());
logger.info("Stack trace: ", t);
} finally {
logger.fatal(msgPrefix + "Process exiting...");
// Category.shutdown();
if (logger instanceof Log4JLogger) {
LogManager.shutdown();
System.exit(0);
* saroMultiProcess.java
* Copyright (c) 2001 bindhu
* All Rights Reserved.
* Fixed Income Banking Products
package com.bindhuw.saro.common;
import static com.bindhuw.saro.infrastructure.util.FileUtility.constructEtcPath;
import static com.bindhuw.saro.infrastructure.util.FileUtility.locate;
import static java.lang.String.format;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.List;
import java.util.Properties;
import org.apache.commons.logging.Log;
import org.xml.sax.SAXException;
import com.bindhuw.saro.infrastructure.dao.saroUtilDao;
import com.bindhuw.saro.infrastructure.exception.saroException;
import com.bindhuw.saro.infrastructure.exception.saroInvalidPropertyException;
import com.bindhuw.saro.infrastructure.file.PasswordFileReader2;
import com.bindhuw.saro.infrastructure.file.PasswordObject;
import com.bindhuw.saro.infrastructure.model.saroMqDefinition;
import com.bindhuw.saro.infrastructure.util.MqUtilities;
import com.bindhuw.saro.infrastructure.util.YorN;
* Models the saro process capable of running multiple threads.
* @author Veera Shivanantham, Anant Kumar Mishra
* @version %I%, %G%
* @see com.bindhuw.saro.common.saroMultiProcess.java
public class saroMultiProcess extends csaroProcess {
// Sleep time for the main thread - 5000 ms (5 sec)
private static final int SLEEP_TIME = 5000;
// Contains the list of saroChildProcess
private List childProcesses;
// flag that holds the request for stopping the process
private boolean alreadyRequestedToStop;
// properties file
private static String propertiesFile;
// debug flag
private static final boolean DEBUG = false;
* Only constructor.
* @param port
* @param envId
* @param processId
* @param termId
* @param processName
public saroMultiProcess(int port, int envId, int processId, String termId, String processName) {
super(port, envId, processId, termId, processName);
this.childProcesses = new ArrayList(10);
* Start all the child processes in the server
public void startServer() {
// // rfs 545
// String killServerOnAllErrors = System.getProperty("KILL_SERVER_ON_ALL_ERRORS");
// if (killServerOnAllErrors != null && killServerOnAllErrors.trim().length() > 0) {
// killServerOnAllErrors = killServerOnAllErrors.trim();
// if (killServerOnAllErrors.startsWith("Y") || killServerOnAllErrors.startsWith("y")
// || killServerOnAllErrors.startsWith("T") || killServerOnAllErrors.startsWith("t")
// || killServerOnAllErrors.startsWith("1"))
// setFlagOn(FLAG_KILL_SERVER_ON_ALL_ERRORS);
for (int i = 0; i < childProcesses.size(); i++) {
saroChildProcess child = (saroChildProcess) childProcesses.get(i);
child.start();
while (this.getStatus() != isaroProcess.SERVER_REQUEST_TO_KILL) {
try {
Thread.sleep(SLEEP_TIME); // sleep for 5 sec.
} catch (InterruptedException ie) {
ie.printStackTrace();
boolean allThreadsDead = true;
for (int i = 0; i < childProcesses.size(); i++) {
saroChildProcess child = (saroChildProcess) childProcesses.get(i);
if (child.isAlive()) {
allThreadsDead = false;
break;
if (allThreadsDead) {
setStatus(isaroProcess.SERVER_REQUEST_TO_KILL);
stopProcess();
* Over rided parent, to check all the child are done and closed properly.
public synchronized void stopProcess() {
if (alreadyRequestedToStop) return;
else alreadyRequestedToStop = true;
// Check all the childs are free. if they are, then stop the childs and exit.
boolean childAlive = true;
while (childAlive) {
childAlive = false;
for (int count = 0; count < childProcesses.size(); count++) {
saroChildProcess child = (saroChildProcess) childProcesses.get(count);
if (child.getServiceStatus() == saroChildProcess.SERVICE_BUSY) {
// If one child is alive no need to check for other childs, go to sleep until childs are
// stopped.
childAlive = true;
if (DEBUG)
System.out.println("[SYSTEM][CHILD]-[" + child.getServiceName() + "]- Alive.");
} else if (child.getServiceStatus() == saroChildProcess.SERVICE_NO_BUSY && child.isAlive()) {
// child thread is sleeping, wake it up to make the exit faster
child.interrupt();
if (childAlive) {
if (DEBUG)
System.out.println("[SYSTEM]-Childs are active, sleeping for " + SLEEP_TIME / 1000
+ " seconds.");
try {
Thread.sleep(SLEEP_TIME); // Sleep for 5 seconds.
} catch (InterruptedException ie) {
if (DEBUG) System.out.println("[SYSTEM]-All child process are finished.");
super.stopProcess();
* Child raised the error. stop all the childs and clean exit of the process.
* @param child
public void raiseClientError(saroChildProcess child) {
System.out.println("[SYSTEM][CHILD]-[" + child.getServiceName() + "]- Child raised the error.");
this.setStatus(isaroProcess.SERVER_REQUEST_TO_KILL);
System.out.println("[SYSTEM] - Closing down all the childs and the system.");
stopProcess();
* Load the properties file from saro_BASE/etc/<process_name>.properties
* @param process name of the process
* @throws FileNotFoundException
* @throws IOException
* @throws saroException
public static void loadPropertiesFor(String process) throws FileNotFoundException, IOException,
saroException {
// try opening it using saro_BASE system property
propertiesFile = System.getProperty("CONFIG_BASE");
if (propertiesFile != null && propertiesFile.trim().length() > 0) {
if (!propertiesFile.trim().endsWith(File.separator))
propertiesFile = propertiesFile.trim() + File.separator;
propertiesFile += ("etc" + File.separator + process + ".properties");
} else {
throw new saroException(" Invalid CONFIG_BASE property: " + propertiesFile
+ ". Unable to locate properties file using this property!");
loadProperties(propertiesFile);
* Loads the properties from the file to the system properties. If the properties value match the password
* file entry pattern the corresponding value from the password file is loaded.
* @param file
public static void loadProperties(String file) throws FileNotFoundException, IOException,
saroInvalidPropertyException, saroException {
propertiesFile = file;
Properties props = new Properties();
props.load(new FileInputStream(propertiesFile));
PasswordFileReader2 passwordFileReader = new PasswordFileReader2();
PasswordObject oPasswordObject = new PasswordObject();
// load the values to the system properties
Enumeration keys = props.keys();
while (keys.hasMoreElements()) {
String key = (String) keys.nextElement();
String value = null;
if (PasswordFileReader2.looksLikePasswordFileKey(props.getProperty(key))) value = passwordFileReader
.getValue(props.getProperty(key)).getValue();
else value = props.getProperty(key);
if (value == null || value.trim().length() == 0)
throw new saroInvalidPropertyException("The value of property " + key
+ " is either null or blank.");
System.setProperty(key, value);
// load DB Properties...
int noOfPools = Integer.parseInt(System.getProperty("NUMBER_OF_POOLS"));
for (int i = 1; i <= noOfPools; i++) {
String[] strURLArray = null;
oPasswordObject = passwordFileReader.getValue(System.getProperty("DBPOOL_DS_NAME_" + i), "SYB",
"URL");
System.setProperty("DBPOOL_SYBASE_URL_" + i, oPasswordObject.getValue());
strURLArray = PasswordFileReader2.splitURL(oPasswordObject.getValue());
System.setProperty("DBPOOL_SERVER_NAME_" + i, strURLArray[3]);
System.setProperty("DBPOOL_PORT_NUMBER_" + i, strURLArray[4]);
oPasswordObject = passwordFileReader.getValue(System.getProperty("DBPOOL_DS_NAME_" + i), "SYB",
"DBN");
System.setProperty("DBPOOL_DB_NAME_" + i, oPasswordObject.getValue());
oPasswordObject = passwordFileReader.getValue(System.getProperty("DBPOOL_DS_NAME_" + i), "SYB",
"USR");
System.setProperty("DB_USER_" + i, oPasswordObject.getValue());
oPasswordObject = passwordFileReader.getValue(System.getProperty("DBPOOL_DS_NAME_" + i), "SYB",
"PWD");
System.setProperty("DB_PASS_" + i, oPasswordObject.getValue());
System.setProperty("DB_USER", System.getProperty("DB_USER_1"));
System.setProperty("DB_PASS", System.getProperty("DB_PASS_1"));
* This method fetches the MQ details from the static table <code>saroMqDefinitions</code> based on the
* <code>processName</code> . For each Mq detail, it then generates mba.xml file only if enabled flag in
* the table is Y and (the mba.xml does not exists for the particular MQ or the boolean createFile flag is
* set to true). Then it loads the child process with argument <code>MqDefinitions</code> and name of
* the class.
* @param logger - For logging reports
* @param processName - MQ definitions are picked based on this process name
* @param serviceName - Name of the service
* @param serviceClass - class of the saroChildProcess3 which we want to run
* @param createNewFile - new mba.xml file is created if true. else mba.xml is created only if not
* present.
* @throws saroInvalidPropertyException
* @throws SQLException
* @throws SAXException
* @throws IOException
* @throws SecurityException
* @throws NoSuchMethodException
* @throws IllegalArgumentException
* @throws InstantiationException
* @throws IllegalAccessException
* @throws InvocationTargetException
protected void loadChildProcesses(Log logger, String processName, String serviceName, Class serviceClass,
boolean createNewFile) throws saroInvalidPropertyException, SQLException, SAXException,
IOException, SecurityException, NoSuchMethodException, IllegalArgumentException,
InstantiationException, IllegalAccessException, InvocationTargetException {
List<saroMqDefinition> list = new saroUtilDao().fetchMqDetails(logger, processName);
for (saroMqDefinition def : list) {
if (def.getEnableFlag() == YorN.Y) {
MqUtilities util = new MqUtilities();
String fileName = serviceName + "." + def.getMqAlias() + ".mba.xml";
String serverFileName = locate(fileName);
if (serverFileName == null || createNewFile) {
serverFileName = constructEtcPath(fileName);
logger.info(format("For mqAlias [%s] writing the file [%s]", def.getMqAlias(),
serverFileName));
String xmlString = util.generateXml(def.toMap());
util.writeFile(serverFileName, xmlString);
} else {
logger.info(format("For mqAlias [%s] using the file [%s]", def.getMqAlias(),
serverFileName));
def.setConfigName(serverFileName);
Constructor constr = serviceClass
.getConstructor(saroMultiProcess.class, saroMqDefinition.class);
saroChildProcess3 w = (saroChildProcess3) constr.newInstance(this, def);
w.setSleepTimeInSecs(def.getSleepTimeInSecs());
w.setServiceActivationStatus(def.getEnableFlag());
getChildProcesses().add(w);
* This method fetches the MQ details from the static table <code>saroMqDefinitions</code> based on the
* <code>processName</code> . For each Mq detail, it then generates mba.xml file only if enabled flag in
* the table is Y and the system property CREATE_NEW_MBA_XML_FILE is set to true. If the property
* CREATE_NEW_MBA_XML_FILE is set to false, then if the mba file already exists, it will not create the
* file. Then it loads the child process with argument <code>MqDefinitions</code> and name of the class.
* @param logger - For logging reports
* @param processName - MQ definitions are picked based on this process name
* @param serviceName - Name of the service
* @param serviceClass - class of the saroChildProcess3 which we want to run
* @throws saroInvalidPropertyException
* @throws SQLException
* @throws SAXException
* @throws IOException
* @throws SecurityException
* @throws NoSuchMethodException
* @throws IllegalArgumentException
* @throws InstantiationException
* @throws IllegalAccessException
* @throws InvocationTargetException
protected void loadChildProcesses(Log logger, String processName, String serviceName, Class serviceClass)
throws saroInvalidPropertyException, SQLException, SAXException, IOException, SecurityException,
NoSuchMethodException, IllegalArgumentException, InstantiationException, IllegalAccessException,
InvocationTargetException {
loadChildProcesses(logger, processName, serviceName, serviceClass, Boolean
.getBoolean("CREATE_NEW_MBA_XML_FILE"));
* Getter for the attribute childProcesses
* @return the list of child processes
public List getChildProcesses() {
return childProcesses;
* Setter for the attribute childProcesses
* @param newChildProcesses
public void setChildProcesses(List newChildProcesses) {
childProcesses = newChildProcesses;
* Getter for the attribute propertiesFile
* @return the value of attribute propertiesFile
public static String getPropertiesFile() {
return propertiesFile;
package com.bindhuw.saro.common;
import com.bindhuw.saro.dbobjects.*;
import com.bindhuw.saro.util.*;
* Title: csaroProcess
* Description:
* Copyright: Copyright (c) 2001
* Company: bindhuW
* @author Veera
* @version 1.0
public class csaroProcess implements isaroProcess{
public static int status = isaroProcess.PROCESS_READY;
public static int serviceCount = 0;
private int controlFlag;
private int loginNumber = 0;
private String userCode = "SYS";
private String loginType = "BACK";
private String hostName;
private int port; //passed In
private int processId; //passed In
private String processName;
private int environmentId; //passed In
private String terminalId; //passed In
private int taskId = 1;
private String longProgramName = "Martini Server";
cUDPService udp;
cUserLoginService userLogin;
public csaroProcess(int port, int envId, int processId, String termId, String processName){
this.setPort(port);
this.setEnvironmentId(envId);
this.processId = processId;
this.terminalId = termId;
this.processName = processName;
//Set the default values.
try{
hostName = java.net.InetAddress.getLocalHost().getHostName();
udp = new cUDPService(this);
udp.setDaemon(true);
userLogin = new cUserLoginService(this);
System.out.println("[SYSTEM]-Starting UDP");
udp.listenToUDP();
System.out.println("[SYSTEM]-Make an entry to User login.");
userLogin.doLogin();
}catch(Exception e) {
System.out.println("[SYSTEM]-Server initialise error.\n"+e.getMessage());
e.printStackTrace();
System.exit(0);
public void finalise() {
userLogin.doLogout();
public void stopProcess() {
System.out.println("[SYSTEM]-Requested to stop this service. Shutting Down System.");
System.out.println("[SYSTEM]-Removing entry from user_login table.");
userLogin.doLogout();
public int getStatus() {
return this.status;
public void setStatus(int status) {
this.status = status;
public int getServiceCount() {
return this.serviceCount;
public String getProcessName() {
return this.processName;
public String getUserCode() {
return this.userCode;
public String getLoginType() {
return this.loginType;
public String getHostName() {
return this.hostName;
public int getPort() {
return this.port;
public int getProcessId() {
return this.processId;
public int getEnvironmentId() {
return this.environmentId;
public String getTerminalId() {
return this.terminalId;
public int getTaskId() {
return this.taskId;
public String getLongProgramName() {
return this.longProgramName;
public int getLoginNumber() {
return loginNumber;
public void setLoginNumber(int loginNum) {
this.loginNumber = loginNum;
public void setHostName(String newHostName) {
hostName = newHostName;
public void setLoginType(String newLoginType) {
loginType = newLoginType;
public void setLongProgramName(String newLongProgramName) {
longProgramName = newLongProgramName;
public void setServiceCount(int newServiceCount) {
this.serviceCount = newServiceCount;
public void setTaskId(int newTaskId) {
taskId = newTaskId;
public void setUserCode(String newUserCode) {
userCode = newUserCode;
public boolean checkService(int serviceId, String client_name) {
if(serviceId == this.SERVICE_START) {
if(this.status == isaroProcess.SERVER_REQUEST_TO_KILL) {
System.out.println("[SYSTEM]-Server flagged to die. No more conections.");
return false;
this.serviceCount ++;
System.out.println("[SYSTEM]-New client connected.("+this.serviceCount+") Clients are connected.");
} else if(serviceId == this.SERVICE_END){
if(this.serviceCount > 0)
this.serviceCount --;
System.out.println("[SYSTEM]-New client connected.("+this.serviceCount+") Clients are connected.");
if(this.status != isaroProcess.SERVER_REQUEST_TO_KILL && this.serviceCount == 0)
setStatus(isaroProcess.PROCESS_READY);
return true;
public void setPort(int newPort) {
port = newPort;
public void setEnvironmentId(int newEnvironmentId) {
environmentId = newEnvironmentId;
public void setProcessId(int newProcessId) {
processId = newProcessId;
public void setProcessName(String newProcessName) {
processName = newProcessName;
public void setTerminalId(String newTerminalId) {
terminalId = newTerminalId;
* Get the value of control flag.
* @return
public int getControlFlag(){
return controlFlag;
* Set the specified flag on.
* @param flag
public void setFlagOn(int flag) {
controlFlag |= flag;
* Set the specified flag off.
* @param flag
public void setFlagOff(int flag) {
controlFlag &= (~flag);
* saroChildProcess3.java
* Copyright (c) 2001 bindhu
* All Rights Reserved.
* Fixed Income Banking Products
package com.bindhuw.saro.common;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.bindhuw.saro.infrastructure.util.YorN;
* Refractored version of the saroChildProcess2. Now uses commons-logging logger.
* @author Veera, Anant
* @version 1.3
* @see com.bindhuw.saro.common.saroChildProcess3.java
public class saroChildProcess3 extends saroChildProcess {
protected Log logger = null;
* Default constructor
public saroChildProcess3() {
super();
logger = LogFactory.getLog(this.getClass());
/** Constructor
* @param newServer Object of saroMultiProcess
public saroChildProcess3(saroMultiProcess newServer) {
this();
setServer(newServer);
* Constructs the object for the server with the specified name
* @param newServer server
* @param name service name
public saroChildProcess3(saroMultiProcess newServer, String name){
this(newServer);
setServiceName(name);
logger = LogFactory.getLog(name);
// set the enabled flag
String enabled = System.getProperty(name + ".enabled", "true");
setServiceActivationStatus(enabled);
// set sleep time
String sleepTimeInSecs = System.getProperty(name + ".sleepTimeInSecs");
if (sleepTimeInSecs != null)
setSleepTimeInSecs(sleepTimeInSecs);
/** Over ride the fundamental behaviour of Runnable target, to acomplish
* the interoprability with saroMultiProcess.
public void run() {
if (getServiceStatus() == SERVICE_DISABLED)
return;
String msgPrefix = "[SYSTEM][" + getServer().getProcessName() + "][" + getServiceName() + "]";
try {
init();
} catch (Exception e) {
setErrorFlag(FATAL_ERROR);
logger.error(msgPrefix + "Error during initialisation of the child process. " + e.getMessage());
logger.info("Stack trace: ", e);
if (getServer().getStatus() != isaroProcess.SERVER_REQUEST_TO_KILL && getErrorFlag() != FATAL_ERROR)
logger.fatal(msgPrefix + "Service launched...");
boolean isAlive = true;
while (isAlive) {
try {
if (getServer().getStatus() == isaroProcess.SERVER_REQUEST_TO_KILL || getErrorFlag() == FATAL_ERROR
|| getErrorFlag() == SERVICE_DONE) {
isAlive = false;
//Set this status flag to SERVICE_NO_BUSY.
setServiceStatus(SERVICE_NO_BUSY);
String msg = null;
if (getErrorFlag() == FATAL_ERROR){
msg = "Terminating thread due to the errors above...";
if((getServer().getControlFlag() & isaroProcess.FLAG_KILL_SERVER_ON_ALL_ERRORS)
== isaroProcess.FLAG_KILL_SERVER_ON_ALL_ERRORS)
getServer().raiseClientError(this);
}else if(getErrorFlag() == SERVICE_DONE){
setErrorFlag(SERVICE_DONE);
msg = "Service completed... ";
} else{
msg = "Request to stop the thread...";
logger.error(msgPrefix + msg);
//Break the loop and exit.
break;
} else {
setServiceStatus(SERVICE_BUSY);
performSpecificService();
setServiceStatus(SERVICE_NO_BUSY);
if (getServer().getStatus() != isaroProcess.SERVER_REQUEST_TO_KILL && getErrorFlag() != FATAL_ERROR) {
try {
Thread.sleep(getSleepTimeInSecs() * 1000); //Sleep for specified time.
} catch (InterruptedException ie) {
// do nothing
} catch (Exception e) {
String msg = "Caught an exception:Message:" + e.getMessage();
logger.error(msgPrefix + msg);
logger.info("Stack trace: ", e);
//Set the error Flag. and set the service status to SERVICE_NO_BUSY.
setErrorFlag(FATAL_ERROR);
getServer().raiseClientError(this);
setServiceStatus(SERVICE_NO_BUSY);
//Raise the error to server.
msg = "Informed to the parent with fatal error. " + "Stoping the child process...";
logger.error(msgPrefix + msg);
logger.fatal(msgPrefix + "Thread exiting... ");
try {
clean();
} catch (Exception e) {
setErrorFlag(FATAL_ERROR);
logger.error(msgPrefix + "Error during clean up of the child process. " + e.getMessage());
logger.info("Stack trace: ", e);
* One time initialization. Called before starting the process.
* @throws Exception
protected void init() throws Exception {
* One time cleanup. Called before exiting the process.
* @throws Exception
protected void clean() throws Exception {
* Getter for property logger
* @return the logger
public Log getLogger() {
return logger;
* Setter for property logger
* @param logger - the logger
public void setLogger(Log logger) {
this.logger = logger;
/** Setter for property sleepTimeInSecs.
* @param sleepTimeInSecs New value of property sleepTimeInSecs
public void setSleepTimeInSecs(String sleepTimeInSecs) {
int sleepTime = getSleepTimeInSecs();
try {
sleepTimeVery interesting.
But who will be willing to read that amount of unformatted code - needless to say, you didn't even post a question...or did I miss it in that pile of characters...? -
How can we control the executions of 4 threads
Hi,
I tried to create 4 threads and tried to run them in custimized manner but i m facing some problem please help me in thi regard.Below is the code which i wrote to schedule the threads in custimized may using wait and notify...
public class MyThreadOne implements Runnable {
String tName;
Thread t;
MyThreadOne(String threadName) {
tName = threadName;
t = new Thread(this, tName);
t.start();
public void run() {
// wait for me to be notified by main thread before continuing.
synchronized (t) {
try {
wait();
} catch(InterruptedException e) {
throw new RuntimeException(e);
// after main thread notifies we run normally.
System.out.println("Thread: " + tName);
System.out.println("Terminating thread: " + tName);
public Thread getThread() {
return this.t;
public static void main(String args[]) {
// Why are the threads start and terminate randomly?
MyThreadOne t1 = new MyThreadOne("1");
MyThreadOne t2 = new MyThreadOne("2");
MyThreadOne t3 = new MyThreadOne("3");
MyThreadOne t4 = new MyThreadOne("4");
// issue a notify to each thread to have them start up
System.out.println(" notify"); //phani
synchronized(t1.getThread()) {
System.out.println("synchronized "); //phani
t1.getThread().notify();
// optional: Thread.sleep(50) here if we want to try to give some lead time...
synchronized(t2.getThread()) {
t2.getThread().notify();
synchronized(t3.getThread()) {
t3.getThread().notify();
synchronized(t4.getThread()) {
t4.getThread().notify();
try {
//NO -> Thread.sleep(10000);
// now wait for each thread to exit.
t1.getThread().join();
t2.getThread().join();
t3.getThread().join();
t4.getThread().join();
// we could also do t1.isAlive();
} catch (InterruptedException e) {
System.out.println(
"Exception: Thread main interrupted.");
System.out.println(
"Terminating thread: main thread.");
Edited by: user13779505 on Jan 18, 2011 3:39 AM
Edited by: user13779505 on Jan 18, 2011 9:23 PM
Edited by: user13779505 on Jan 19, 2011 11:39 PMuser13779505 wrote:
if there is some correction in my program please correct itThats hard to do as I assume you doing this as a learning exercise, meaning you have make the changes.
I have already said the best way is to use a loop.
When you get the code working it should do the same thing as the following. Why would you make it more complicated?
for(int tName=1;tName<=4;tname++)
System.out.println("Thread: " + tName + "\nTerminating thread: " + tName);
System.out.println("Terminating thread: main thread."); -
Non-terminating console with JOptionPane
This short program...
import javax.swing.*;
public class Test {
public static void main(String[] args) {
JOptionPane.showMessageDialog(null, "Hello!");
...running under Windows 2000 leaves the console open after the message dialog closes. I know I can terminate the program with System.exit(0), but this doesn't seem to be the "correct" approach. Some cleanup function may be required to dispose of the latent message window, but I have not found it yet. Any idea?
Thanks!Hey Dan, thanks for your reply! You may be correct regarding non-terminating threads being responsible for the freeze. I have modified the test program to display running threads before and after the dialog box is shown:
import javax.swing.*;
public class Test {
public static void main(String[] args)
// Show all threads before the dialog box:
Thread[] array = new Thread[20];
int count = Thread.enumerate(array);
for (int i = 0; i < count; i++) {
System.out.println("Thread " + i + ": " + array);
System.out.println();
// Create the dialog box:
JOptionPane.showMessageDialog(null, "Hello!");
// Show all threads after the dialog box:
count = Thread.enumerate(array);
for (int i = 0; i < count; i++) {
System.out.println("Thread " + i + ": " + array[i]);
System.out.println();
Result --> there is a single "main" thread before:
Thread 0: Thread[main,5,main]
...but there are FIVE MORE threads sitting there after:
Thread 0: Thread[main,5,main]
Thread 1: Thread[AWT-Shutdown,5,main]
Thread 2: Thread[AWT-Windows,6,main]
Thread 3: Thread[Image Fetcher 0,8,main]
Thread 4: Thread[Java2D Disposer,10,main]
Thread 5: Thread[AWT-EventQueue-0,6,main]
One of these is called "AWT-Shutdown" and you would thing it would do its thing and shut down AWT but that does not happen. Another is "Java2D Disposer" and it doesn't seem to dispose of what is necessary either.
Do you have any additional insight into this situation? -
Detecting (And Killing) Abandoned Threads
Hi All,
I'm going to try to make my question as simple as possible: I'm trying to use the service guardian to automatically detect an abandoned thread and kill the originating process. So basically I set up a test that ends up with an abandoned thread and configured service Guardian... but it didn't work :D
What I first did is setting the guardian's timeout to be very short (6 seconds). When I do that, what I get is:
2010-03-26 10:03:29.389/0.859 Oracle Coherence GE 3.5.3/465 <Info> (thread=main, member=n/a): Loaded cache configuration from "file:/C:/workspaces/POC/InvocableTest/bin/coherence-cache-config.xml"
2010-03-26 10:03:29.952/1.422 Oracle Coherence GE 3.5.3/465 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2010-03-26 10:03:35.295/6.765 Oracle Coherence GE 3.5.3/465 <Error> (thread=Cluster, member=n/a): Attempting recovery (due to soft timeout) of Guard{Daemon=TcpRingListener}
2010-03-26 10:03:35.904/7.374 Oracle Coherence GE 3.5.3/465 <Error> (thread=Cluster, member=n/a): Terminating guarded execution (due to hard timeout) of Guard{Daemon=TcpRingListener}
Coherence <Error>: Halting JVM due to unrecoverable service failure
2010-03-26 10:03:36.904/8.374 Oracle Coherence GE 3.5.3/465 <Error> (thread=Termination Thread, member=n/a): Full Thread Dump
ThreadCluster
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Unknown Source)
So that's before the thread gets abandoned... it seems that it detects a deadlock when starting coherence. But that's ok, I know the service guardian is working. The problem comes when I set the timeout to something more realistic: 35 seconds for instance. I run my test, the thread gets to an abandoned state and the service guardian does not see it :(
Any Ideas?
h4. My override config is:
<coherence>
<cluster-config>
<service-guardian>
<timeout-milliseconds system-property="tangosol.coherence.guard.timeout">6000</timeout-milliseconds>
<service-failure-policy>exit-process</service-failure-policy>
</service-guardian>
</cluster-config>
<logging-config>
<severity-level system-property="tangosol.coherence.log.level">5</severity-level>
<character-limit system-property="tangosol.coherence.log.limit">0</character-limit>
</logging-config>
</coherence>
Thanks!
Fernando
Edited by: ZeoS on Mar 26, 2010 7:13 AMit's an Invovable running on the server.
The description in the debugger is: InvocationServiceWorker
Maybe this helps:
10-03-26 12:03:41.279/72.663 Oracle Coherence GE 3.5.3/465 <Error> (thread=Abandon, member=1): A worker thread "InvocationServiceWorker:0 executing task "Message "InvocationRequest"
FromMember=Member(Id=1, Timestamp=2010-03-26 12:02:29.866, Address=10.31.106.86:8088, MachineId=32598, Location=site:someSite.com,machine:ZeoSWorkStation,process:1084, Role=TestInvocableExpiration)
FromMessageId=0
Internal=false
MessagePartCount=0
PendingCount=0
MessageType=1
ToPollId=0
Poll=null
Packets
Service=InvocationService{Name=InvocationService, State=(SERVICE_STARTED), Id=3, Version=3.1, OldestMemberId=1}
ToMemberSet=MemberSet(Size=1, BitSetCount=2, ids=[1])
NotifySent=false
InvocationRequest{Query, Task=TestInvocableExpiration.SocketInvocable@16b352c}
}", did not respond to 8 interrupt requests. The execution was canceled. The thread is abandoned...
Fernando -
Hi,
I'm running 1.4.2 (Hotspot) on Mac OS X 10.3.3 Server. I'm getting OutOfMemory exceptions when I've initialized several instances of our web application running in Tomcat (latest stable 4.x). This happens even though more than 50% of the memory allocated to the JVM is free and there are plenty of processes before the ulimit caps are met. Any light on this subject would be MOST appreciated.
The system settings are as follows:
Memory usage at the time of the error (printed from vm Runtime):
Memory freeMemory : 495268096
Total Memory : 725876736
Java argument commands:
-Xms700mb -Xmx700mb
ulimit settings:
core file size (blocks, -c) 0
data seg size (kbytes, -d) 6144
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 10000
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 65536
cpu time (seconds, -t) unlimited
max user processes (-u) 500
virtual memory (kbytes, -v) 71680
Current Threads: 84
Output:
java.lang.OutOfMemoryError
May 5, 2004 5:32:26 PM
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception executing
org.apache.jk.common.SocketConnection@60d45b, terminating thread
java.lang.OutOfMemoryError
May 5, 2004 5:33:33 PM
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception executing
org.apache.jk.common.SocketConnection@c7af15, terminating thread
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
The following error report describes the Java stack information at the
crash.
The native stack information can be found in the crash log generated by
CrashReporter.
An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : Bus Error occurred at PC=0x815A2C54
Function=[Unknown.]
Library=(N/A)
NOTE: We are unable to locate the function name symbol for the error
just occurred. Please refer to release documentation for possible
reason and solutions.
Current Java thread:
at sun.misc.Unsafe.defineClass(Native Method)
at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
at
sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:
381)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.jav
a:377)
at
sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerat
or.java:59)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:28)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at
com.attask.timeline.DateMethod.setStartDate(DateMethod.java:657)
at com.attask.timeline.DateMethod.resetNode(DateMethod.java:269)
at
com.attask.timeline.DateMethod.startCalculate(DateMethod.java:95)
at com.attask.timeline.Timeline.calculate(Timeline.java:207)
at com.attask.timeline.Timeline.calculate(Timeline.java:202)
at
com.attask.AtProjects.calculateTimelines(AtProjects.java:2256)
at
com.attask.AtTasks.initiateTimelineUpdateEvents(AtTasks.java:2506)
at com.attask.AtTasks.addTxn(AtTasks.java:2588)
at com.attask.AtBean.addTxn(AtBean.java:381)
at com.attask.AtBean.add(AtBean.java:330)
at com.attask.AtBean.add(AtBean.java:372)
at org.apache.jsp.addTask_jsp._jspService(addTask_jsp.java:192)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja
va:210)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:
295)
at
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:247)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:193)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:256)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i
nvokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:
480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:191)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i
nvokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:
480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:
2417)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:180)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i
nvokeNext(StandardPipeline.java:643)
at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherVa
lve.java:171)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i
nvokeNext(StandardPipeline.java:641)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
:172)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i
nvokeNext(StandardPipeline.java:641)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:
480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
java:174)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.i
nvokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:
480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:193)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:309)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:387)
at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:673)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:
615)
at
org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:786)
Dynamic libraries and executable:
[0x00003acc-0x0000643c] /Library/Java/Home/bin/java
[0x90df0f2c-0x90df36c4]
/System/Library/Frameworks/JavaVM.framework/Versions/A/JavaVM
[0x90190b98-0x9022a5cc]
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/
CoreFoundation
[0x90000e40-0x900fab40] /usr/lib/libSystem.B.dylib
[0x909f1498-0x90b168f8]
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
[0x968d14ac-0x969a3b7c] /usr/lib/libicucore.A.dylib
[0x939d0eb4-0x939d257c] /usr/lib/system/libmathCommon.A.dylib
[0x94a30ad8-0x94adc678] /usr/lib/libxml2.2.dylib
[0x90700eb4-0x90700f64]
/System/Library/Frameworks/CoreServices.framework/Versions/A/
CoreServices
[0x90d008f4-0x90d166f0]
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/
SystemConfiguration
[0x908311e0-0x9083de7c] /usr/lib/libobjc.A.dylib
[0x945b0834-0x945b857c] /usr/lib/libz.1.dylib
[0x96cb0780-0x96cc4aa8] /usr/lib/libiconv.2.dylib
[0x9028114c-0x90341010]
/System/Library/Frameworks/CoreServices.framework/Versions/A/
Frameworks/CarbonCore.framework/Versions/A/CarbonCore
[0x90610de0-0x906796c8]
/System/Library/Frameworks/CoreServices.framework/Versions/A/
Frameworks/OSServices.framework/Versions/A/OSServices
[0x927413cc-0x92772154]
/System/Library/Frameworks/CoreServices.framework/Versions/A/
Frameworks/CFNetwork.framework/Versions/A/CFNetwork
[0x940613b8-0x94074d28]
/System/Library/Frameworks/CoreServices.framework/Versions/A/
Frameworks/WebServicesCore.framework/Versions/A/WebServicesCore
[0x94651134-0x9469e0b8]
/System/Library/Frameworks/CoreServices.framework/Versions/A/
Frameworks/SearchKit.framework/Versions/A/SearchKit
[0x90571080-0x905cc1f4]
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
[0x907f1260-0x907f8ac4]
/System/Library/PrivateFrameworks/DiskArbitration.framework/Versions/A/
DiskArbitration
[0x920c1414-0x9223269c]
/System/Library/Frameworks/Security.framework/Versions/A/Security
[0x94610d1c-0x94627514] /usr/lib/libresolv.9.dylib
[0x910b0e40-0x910f34b8]
/System/Library/PrivateFrameworks/Bom.framework/Versions/A/Bom
[0x81360a20-0x815793f0]
/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Libraries/
libserver.dylib
[0x8e282b6c-0x8e28b214]
/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Libraries/
libverify.dylib
[0x80f31600-0x80f477c4]
/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Libraries/
libjava.jnilib
[0x8e27275c-0x8e27c2c8]
/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Libraries/
libzip.jnilib
[0x8a4cbf08-0x8a4d3c7c]
/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Libraries/
libnet.jnilib
[0x8a4d8fe8-0x8a4dc46c]
/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Libraries/
libnio.jnilib
Local Time = Wed May 5 17:34:44 2004
Elapsed Time = 57237517
# The exception above was detected in native code outside the VM
# Java VM: Java HotSpot(TM) Client VM (1.4.2-34 mixed mode)Brilliant! The MaxPermSize seems to have made a
difference. The Apple implementation defaults to 32mb.What makes your application to exceed the default Perm Size ? Even I had the same problem but I am still trying to verify its cause.
Referring to some docs on Net I have read all the Class and Static variables does get stored in the Method Area(probabily the PermSpace) . So if your application running have
1) Larger Number of classes getting loaded at RunTime the Perm will grow and will thus result in the some weild error.
2) Or the static variable in loaded class in the MEthod Area grows dynamically .I am not able to see the results with this , this can be wrong also .waiting for some one who know more to clear this .
Hope this help you .
Regards
Vicky -
Is it a Limitation of Named Cache Storage- Fails for large volume ???
I debugged the code which loads data from database into the cache as mentioned in the posting : Pre-loading the Cache from Database during application start-up
Now what this code does is load 869 rows from database into java.util.Map using Hibernate loadAll() method. All is fine uptill this point.
The next step is to putAll the entries into cache i.e. contactCache.putAll(buffer). This is where it hungs for a min and i see org.eclipse.jdi.TimeoutException followed by below exception stack trace
IN DEFAULT CACHE SERVER JVM
2009-10-30 10:53:44.076/1342.849 Oracle Coherence GE 3.5.2/463 <Warning> (thread=PacketPublisher, member=1): Experienced a 1390 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2009-10-30 10:31:54.697, Address=165.137.250.122:8089, MachineId=54906, Location=site:cable.comcast.com,machine:PACDCL-CJWWND1b,process:4856); 23 packets rescheduled, PauseRate=0.0010, Threshold=2080
2009-10-30 11:06:10.060/2088.833 Oracle Coherence GE 3.5.2/463 <Error> (thread=Cluster, member=1): Attempting recovery (due to soft timeout) of Guard{Daemon=DistributedCache}
2009-10-30 11:06:12.430/2091.203 Oracle Coherence GE 3.5.2/463 <Error> (thread=Cluster, member=1): Terminating guarded execution (due to hard timeout) of Guard{Daemon=DistributedCache}
2009-10-30 11:06:15.657/2094.430 Oracle Coherence GE 3.5.2/463 <Info> (thread=Main Thread, member=1): Restarting Service: DistributedCache
2009-10-30 11:06:15.954/2094.727 Oracle Coherence GE 3.5.2/463 <Error> (thread=Main Thread, member=1): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: DistributedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=129, BackupPartitions=128}
Coherence <Error>: Halting this cluster node due to unrecoverable service failure
2009-10-30 11:06:16.671/2095.444 Oracle Coherence GE 3.5.2/463 <Error> (thread=Termination Thread, member=1): Full Thread Dump
Thread[Cluster|Member(Id=1, Timestamp=2009-10-30 10:31:31.621, Address=165.137.250.122:8088, MachineId=54906, Location=site:cable.comcast.com,machine:PACDCL-CJWWND1b,process:5380),5,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:595)
Thread[(Code Generation Thread 1),5,system]
Thread[(Signal Handler),5,system]
Thread[TcpRingListener,6,Cluster]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
java.net.ServerSocket.implAccept(ServerSocket.java:450)
java.net.ServerSocket.accept(ServerSocket.java:421)
com.tangosol.coherence.component.net.socket.TcpSocketAccepter.accept(TcpSocketAccepter.CDB:18)
com.tangosol.coherence.component.util.daemon.TcpRingListener.acceptConnection(TcpRingListener.CDB:10)
com.tangosol.coherence.component.util.daemon.TcpRingListener.onNotify(TcpRingListener.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:595)
Thread[PacketSpeaker,8,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
com.tangosol.coherence.component.util.queue.ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketSpeaker.onNotify(PacketSpeaker.CDB:62)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:595)
Thread[PacketPublisher,6,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onWait(PacketPublisher.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:595)
Thread[(VM Periodic Task),10,system]
Thread[(Sensor Event Thread),5,system]
Thread[(Attach Listener),5,system]
Thread[(GC Main Thread),5,system]
Thread[(Code Optimization Thread 1),5,system]
Thread[Invocation:Management:EventDispatcher,5,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:595)
Thread[Main Thread,5,main]
java.lang.Object.wait(Native Method)
com.tangosol.net.DefaultCacheServer.main(DefaultCacheServer.java:79)
Thread[Logger@9265725 3.5.2/463,3,main]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:595)
Thread[Invocation:Management,5,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:9)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:595)
Thread[Reference Handler,10,system]
java.lang.ref.Reference.getPending(Native Method)
java.lang.ref.Reference.access$000(Unknown Source)
java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Thread[PacketListenerN,8,Cluster]
java.net.PlainDatagramSocketImpl.receive0(Native Method)
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
java.net.DatagramSocket.receive(DatagramSocket.java:712)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:595)
Thread[Finalizer,8,system]
java.lang.Thread.run(Thread.java:595)
Thread[DistributedCache,5,Cluster]
com.tangosol.util.Binary.<init>(Binary.java:87)
com.tangosol.util.Binary.<init>(Binary.java:61)
com.tangosol.io.AbstractByteArrayReadBuffer.toBinary(AbstractByteArrayReadBuffer.java:152)
com.tangosol.io.pof.PofBufferReader.readBinary(PofBufferReader.java:3412)
com.tangosol.io.pof.PofBufferReader.readAsObject(PofBufferReader.java:2854)
com.tangosol.io.pof.PofBufferReader.readObject(PofBufferReader.java:2600)
com.tangosol.io.pof.ConfigurablePofContext.deserialize(ConfigurablePofContext.java:348)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service.readObject(Service.CDB:4)
com.tangosol.coherence.component.net.Message.readObject(Message.CDB:1)
com.tangosol.coherence.component.net.message.requestMessage.distributedCacheRequest.MapRequest.read(MapRequest.CDB:24)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:123)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onNotify(DistributedCache.CDB:3)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:595)
Thread[PacketReceiver,7,Cluster]
java.lang.Object.wait(Native Method)
com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onWait(PacketReceiver.CDB:2)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
java.lang.Thread.run(Thread.java:595)
Thread[PacketListener1,8,Cluster]
java.net.PlainDatagramSocketImpl.receive0(Native Method)
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
java.net.DatagramSocket.receive(DatagramSocket.java:712)
com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:595)
Thread[Termination Thread,5,Cluster]
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Thread.java:1434)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:585)
com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:791)
com.tangosol.coherence.component.net.Cluster.onServiceFailed(Cluster.CDB:5)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard.terminate(Grid.CDB:17)
com.tangosol.net.GuardSupport$2.run(GuardSupport.java:652)
java.lang.Thread.run(Thread.java:595)
2009-10-30 11:06:20.958/2099.731 Oracle Coherence GE 3.5.2/463 <Info> (thread=Main Thread, member=1): Restarting Service: DistributedCache
2009-10-30 11:06:20.958/2099.731 Oracle Coherence GE 3.5.2/463 <Error> (thread=Main Thread, member=1): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: DistributedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=129, BackupPartitions=128}
2009-10-30 11:07:17.682/2156.455 Oracle Coherence GE 3.5.2/463 <Info> (thread=Main Thread, member=1): Restarting Service: DistributedCache
2009-10-30 11:07:17.682/2156.455 Oracle Coherence GE 3.5.2/463 <Error> (thread=Main Thread, member=1): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: DistributedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=129, BackupPartitions=128}
2009-10-30 11:07:17.682/2156.455 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): TcpRing: disconnected from member 2 due to a kill request
2009-10-30 11:07:17.682/2156.455 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 2 left service Management with senior member 1
2009-10-30 11:07:17.682/2156.455 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 2 left service DistributedCache with senior member 1
2009-10-30 11:07:17.682/2156.455 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2009-10-30 11:07:17.682, Address=165.137.250.122:8089, MachineId=54906, Location=site:cable.comcast.com,machine:PACDCL-CJWWND1b,process:4856) left Cluster with senior member 1
2009-10-30 11:07:17.682/2156.455 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Service guardian is 51795ms late, indicating that this JVM may be running slowly or experienced a long GC
2009-10-30 11:07:18.073/2156.846 Oracle Coherence GE 3.5.2/463 <Info> (thread=PacketListenerN, member=1): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2009-10-30 11:07:22.696/2161.469 Oracle Coherence GE 3.5.2/463 <Error> (thread=Cluster, member=1): Attempting recovery (due to soft timeout 26277ms ago) of Guard{Daemon=TcpRingListener}
2009-10-30 11:07:22.696/2161.469 Oracle Coherence GE 3.5.2/463 <Info> (thread=Main Thread, member=1): Restarting Service: DistributedCache
2009-10-30 11:07:22.696/2161.469 Oracle Coherence GE 3.5.2/463 <Error> (thread=Main Thread, member=1): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: DistributedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=129, BackupPartitions=128}
2009-10-30 11:07:26.835/2165.608 Oracle Coherence GE 3.5.2/463 <Info> (thread=PacketListenerN, member=1): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2009-10-30 11:07:27.709/2166.482 Oracle Coherence GE 3.5.2/463 <Info> (thread=Main Thread, member=1): Restarting Service: DistributedCache
2009-10-30 11:07:27.709/2166.482 Oracle Coherence GE 3.5.2/463 <Error> (thread=Main Thread, member=1): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: DistributedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=129, BackupPartitions=128}
2009-10-30 11:07:32.723/2171.496 Oracle Coherence GE 3.5.2/463 <Info> (thread=Main Thread, member=1): Restarting Service: DistributedCache
2009-10-30 11:07:32.723/2171.496 Oracle Coherence GE 3.5.2/463 <Error> (thread=Main Thread, member=1): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: DistributedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=129, BackupPartitions=128}
2009-10-30 11:07:42.796/2181.569 Oracle Coherence GE 3.5.2/463 <Info> (thread=Main Thread, member=1): Restarting Service: DistributedCache
2009-10-30 11:07:42.843/2181.616 Oracle Coherence GE 3.5.2/463 <Error> (thread=Main Thread, member=1): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: DistributedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=129, BackupPartitions=128}
2009-10-30 11:07:42.890/2181.663 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Service guardian is 10089ms late, indicating that this JVM may be running slowly or experienced a long GC
2009-10-30 11:07:42.968/2181.741 Oracle Coherence GE 3.5.2/463 <Info> (thread=PacketListenerN, member=1): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2009-10-30 11:07:47.857/2186.630 Oracle Coherence GE 3.5.2/463 <Info> (thread=Main Thread, member=1): Restarting Service: DistributedCache
2009-10-30 11:07:47.935/2186.708 Oracle Coherence GE 3.5.2/463 <Error> (thread=Main Thread, member=1): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: DistributedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=129, BackupPartitions=128}
2009-10-30 11:07:50.527/2189.300 Oracle Coherence GE 3.5.2/463 <Info> (thread=PacketListenerN, member=1): Scheduled senior member heartbeat is overdue; rejoining multicast group.
2009-10-30 11:07:52.948/2191.721 Oracle Coherence GE 3.5.2/463 <Info> (thread=Main Thread, member=1): Restarting Service: DistributedCache
2009-10-30 11:07:52.948/2191.721 Oracle Coherence GE 3.5.2/463 <Error> (thread=Main Thread, member=1): Failed to restart services: java.lang.IllegalStateException: Failed to unregister: DistributedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=129, BackupPartitions=128}
- SQL Error: 1400, SQLState: 23000
- ORA-01400: cannot insert NULL into ("CTXOWNER"."CTX_TRM_TXTS"."CTX_TRM_TXT_ID")
- SQL Error: 1400, SQLState: 23000
- ORA-01400: cannot insert NULL into ("CTXOWNER"."CTX_TRM_TXTS"."CTX_TRM_TXT_ID")
Coherence <Error>: Halting this cluster node due to unrecoverable service failureNow i do see its complaining about cannot insert null values. But wondering how come it was able to insert from database into java.util.Map. Its matter of dumping from Map to Coherence Cache which is another Map
IN CACHE FACTORY VM
Map (com.comcast.customer.contract.contract.hibernate.Term):
2009-10-30 11:06:46.076/2095.134 Oracle Coherence GE 3.5.2/463 <Warning> (thread=PacketPublisher, member=2): Timeout while delivering a packet; requesting the departure confirmation for Member(Id=3, Timestamp=2009-10-30 10:52:20.758, Address=165.137.250.122:8090, MachineId=54906, Location=site:cable.comcast.com,machine:PACDCL-CJWWND1b,process:2756)
by MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2009-10-30 10:31:31.621, Address=165.137.250.122:8088, MachineId=54906, Location=site:cable.comcast.com,machine:PACDCL-CJWWND1b,process:5380)
Map (com.comcast.customer.contract.contract.hibernate.Term):
Map (com.comcast.customer.contract.contract.hibernate.Term): 2009-10-30 11:06:46.887/2095.945 Oracle Coherence GE 3.5.2/463 <Error> (thread=PacketPublisher, member=2): This node appears to have become disconnected from the rest of the cluster containing 2 nodes. All departure confirmation requests went unanswered.
Stopping cluster service.
Map (com.comcast.customer.contract.contract.hibernate.Term): 2009-10-30 11:06:48.773/2097.831 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=2): Service Cluster left the cluster
2009-10-30 11:06:49.257/2098.315 Oracle Coherence GE 3.5.2/463 <D5> (thread=Invocation:Management, member=2): Service Management left the cluster
2009-10-30 11:06:49.257/2098.315 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=2): Service DistributedCache left the clusterIN JUnit Test VM
Coherence <Error>: Halting this cluster node due to unrecoverable service failurePlease note i am running Default Cache Server VM, Cache Factory VM, Eclipse JUnit Test VM in the same machine.
Please note the same piece of code works absolutely fine when i load other object which return 154 rows.Thanks for quick response.
>
So using the local scheme, you place 869 objects into that cache, correct? Does that work?
I didn't tried with local scheme. But i did try with <read-write-backing-map> scheme as it was giving problem i reduced the size to 100 & changed to local-scheme.
If you would like me to try with local-scheme i would do so but it will not prove anything as we need Hibernate Cache store to do write's
>
Can you explain what the remaining issue is? (What part is failing?)
There are several issues and i am really striving to make it work :)-
Here is the list
- revert back to <read-write-backing-map> scheme so that i can pre-poulate the cache from database so that subsequent reads and writes hit the cache instead of database
- to pre-populate the cache during application start-up . We use Spring 2.5, Hibernate 3.2
- the queryContract(contract) method is similar to Search screen i.e. it takes sample contract object as an argument with some attributes populated. I am using Filter API to return the List of Contract objects based on the search parameters of sample contract as follows
Filter filter = new EqualsFilter(IdentityExtractor.INSTANCE, contract);
Set setEntries = contractCache.entrySet(filter); The above code expects all the attributes of sample contract object are fully populated and if not it throws Null Pointer Exception
For example if date attribute is null then Null Pointer Exception is thrown at the following line
writer.writeLong(2, this.date.getTimeInMillis());
I greatly appreciate the inventor of Tangosol Coherence product responding to my queries on the forum. Hopefully with his help i will be able to resolve these issues :)- -
How to properly implement custom ServiceFailurePolicy
I'd like to use Coherence service guardian to interrupt & cancel hung cacheloader threads, which are going into infinite loop due to database connection became unresponsive.
For that purpose I've tried first to create working testcase in order to learn how service guardian works.
I've created a basic distributed cache with fake cache loader to imitate a delay in response, then an infinite loop, and custom test service guardian policy.
I do see cache loader being interrupted by a call to onGuardableRecovery, but then strange thing happens: despite cacheloader thread being successfully released, service guardian still thinks that the service is stuck and tries several times to recover by calling onGuardableRecovery and finally to terminate service with the call to onGuardableTerminate.
What I'm missing here or doing wrong ?
Below are testcase classes & cache config:
package test.guardian;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.cache.CacheStore;
import com.tangosol.util.Base;
import java.util.Collection;
import java.util.Map;
public class testCacheStore extends Base implements CacheStore {
private String datasource_name = null;
public static volatile boolean loop_thread = true;
public testCacheStore(String p_datasource_name) {
CacheFactory.log("inside cacheloader constructor, p_datasource_name="+p_datasource_name,CacheFactory.LOG_DEBUG);
this.datasource_name = p_datasource_name;
public Object load(Object oKey) {
CacheFactory.log("inside cacheloader",CacheFactory.LOG_DEBUG);
try {
Thread.sleep(120*1000);
catch (InterruptedException e) {
CacheFactory.log("loader thread was interrupted"+"\n"+CacheFactory.getStackTrace(e),CacheFactory.LOG_DEBUG);
while (testCacheStore.loop_thread) {String n = "1"+"2";}
CacheFactory.log("loader stop_thread flag was set to stop thread",CacheFactory.LOG_DEBUG);
CacheFactory.log("return value from the store",CacheFactory.LOG_DEBUG);
return ((String) oKey)+"-value";
public Map loadAll(Collection colKeys) {
throw new UnsupportedOperationException();
public void store(Object oKey, Object oValue) {
public void storeAll(Map mapEntries) {
throw new UnsupportedOperationException();
public void erase(Object oKey) {
public void eraseAll(Collection colKeys) {
CacheFactory.log("inside eraseAll method", CacheFactory.LOG_DEBUG);
throw new UnsupportedOperationException();
package test.guardian;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.Guardable;
import com.tangosol.net.ServiceFailurePolicy;
import com.tangosol.net.Service;
import com.tangosol.net.Cluster;
public class mycustomGPolicy implements ServiceFailurePolicy {
public static mycustomGPolicy getInstance() {
return new mycustomGPolicy();
public mycustomGPolicy(String p_info) {
CacheFactory.log("parametrized policy class instance created",CacheFactory.LOG_DEBUG);
CacheFactory.log("p_info="+p_info,CacheFactory.LOG_DEBUG);
public mycustomGPolicy() {
CacheFactory.log("policy class instance created",CacheFactory.LOG_DEBUG);
//CacheFactory.log("p_info="+p_info,CacheFactory.LOG_DEBUG);
public void onGuardableRecovery(Guardable p_guardable, Service p_service) {
CacheFactory.log("inside onGuardableRecovery",CacheFactory.LOG_DEBUG);
CacheFactory.log("p_guardable="+p_guardable.toString(),CacheFactory.LOG_DEBUG);
CacheFactory.log("p_guardable.getClass().getCanonicalName()="+p_guardable.getClass().getCanonicalName(),CacheFactory.LOG_DEBUG);
testCacheStore.loop_thread = false;
p_guardable.recover();
p_guardable.getContext().heartbeat();
public void onGuardableTerminate(Guardable p_guardable, Service p_service) {
CacheFactory.log("inside onGuardableTerminate",CacheFactory.LOG_DEBUG);
CacheFactory.log("p_guardable="+p_guardable.toString(),CacheFactory.LOG_DEBUG);
CacheFactory.log("p_guardable.getClass().getCanonicalName()="+p_guardable.getClass().getCanonicalName(),CacheFactory.LOG_DEBUG);
testCacheStore.loop_thread = false;
p_guardable.terminate();
public void onServiceFailed(Cluster p_cluster) {
CacheFactory.log("inside onServiceFailed",CacheFactory.LOG_DEBUG);
package test.guardian;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.NamedCache;
public class testGuardianCache {
public static void main(String[] args) {
try {
log("building cache instance");
NamedCache cache = CacheFactory.getCache("test-guardable");
log("requesting cache data");
String resp4 = (String)cache.get("test1223g");
log("response:");
System.out.println(resp4+"\n");
} catch (com.tangosol.io.pof.PortableException e) {
Throwable e_src = e;
while (e_src.getCause() != null ) {
e_src = e_src.getCause();
e_src.printStackTrace();
public static void log(String info) {
System.out.println(info);
<?xml version="1.0" encoding="UTF-8"?>
<cache-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>test-guardable</cache-name>
<scheme-name>test-guardable-schema</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>test-guardable-schema</scheme-name>
<service-name>test-guardable-service</service-name>
<backing-map-scheme>
<read-write-backing-map-scheme>
<cachestore-timeout>0</cachestore-timeout>
<internal-cache-scheme>
<external-scheme>
<scheme-name>LocalNioLtd</scheme-name>
<high-units>10000</high-units>
<unit-calculator>FIXED</unit-calculator>
<nio-memory-manager>
<initial-size>10M</initial-size>
<maximum-size>50M</maximum-size>
</nio-memory-manager>
</external-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>test.guardian.testCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>jdbc-coherence/MY-CACHE-DS</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<rollback-cachestore-failures>true</rollback-cachestore-failures>
<read-only>true</read-only>
<write-delay-seconds>0</write-delay-seconds>
</read-write-backing-map-scheme>
</backing-map-scheme>
<listener/>
<autostart>true</autostart>
<thread-count>10</thread-count>
<guardian-timeout>10000</guardian-timeout>
<service-failure-policy>
<class-name>test.guardian.mycustomGPolicy</class-name>
<init-params>
<init-param>
<param-name>datasource</param-name>
<param-type>java.lang.String</param-type>
<param-value>jdbc-coherence/MY-CACHE-DS</param-value>
</init-param>
</init-params>
</service-failure-policy>
</distributed-scheme>
</caching-schemes>
</cache-config>
<?xml version="1.0"?>
<coherence>
<services>
<service>
<service-type>DistributedCache</service-type>
<service-component>DistributedCache</service-component>
<init-params>
<init-param id="4">
<param-name>local-storage</param-name>
<param-value system-property="tangosol.coherence.distributed.localstorage">false</param-value>
</init-param>
</init-params>
</service>
</services>
<cluster-config>
<member-identity>
<cluster-name>WORKSTATION</cluster-name>
</member-identity>
<unicast-listener>
<address system-property="tangosol.coherence.localhost">localhost</address>
<port system-property="tangosol.coherence.localport">7070</port>
<port-auto-adjust system-property="tangosol.coherence.localport.adjust">true</port-auto-adjust>
<well-known-addresses>
<socket-address id="1"><address>localhost</address><port>7070</port></socket-address>
</well-known-addresses>
</unicast-listener>
<authorized-hosts/>
<logging-config>
<severity-level system-property="tangosol.coherence.log.level">5</severity-level>
<character-limit system-property="tangosol.coherence.log.limit">0</character-limit>
</logging-config>
</cluster-config>
<logging-config>
<severity-level>5</severity-level>
<message-format>{date} <{level}> (thread={thread}, member={member}): {text}</message-format>
<character-limit>8192</character-limit>
</logging-config>
</coherence>
setlocal
set COHERENCE_HOME=c:\coherence36
set PROG_HOME=C:\coherence36\deploy
set COH_OPTS=-cp %COHERENCE_HOME%\lib\coherence.jar;%PROG_HOME%\testguardable.jar
set JMX_OPTS=-Dcom.sun.management.jmxremote -Dtangosol.coherence.management=all -Dtangosol.coherence.management.remote=true -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false
set OVERRIDE_OPTS=-Dtangosol.coherence.override=%PROG_HOME%\tangosol-coherence-override.xml
rem
rem
start "1testguardable" /MAX C:\jdev10g\jdk\jre\bin\java.exe %COH_OPTS% %OVERRIDE_OPTS% -Xms256m -Xmx256m -Xloggc: -Dtangosol.coherence.distributed.localstorage=true -Dtangosol.coherence.cacheconfig=%PROG_HOME%\testguardable-cache-config.xml com.tangosol.net.DefaultCacheServerINFO - 2010-07-23 23:43:07.906 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/C:/coherence36/lib/coherence.jar!/tangosol-coherence.xml"
INFO - 2010-07-23 23:43:07.921 <Info> (thread=main, member=n/a): Loaded operational overrides from "file:/C:/-PRG/coherence_ws/Guardian/deploy/tangosol-coherence-override.xml"
DEBUG - 2010-07-23 23:43:07.937 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
DEBUG -
Oracle Coherence Version 3.6.0.0 Build 17229
Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
INFO - 2010-07-23 23:43:08.531 <Info> (thread=main, member=n/a): Loaded cache configuration from "file:/C:/-PRG/coherence_ws/Guardian/deploy/testguardable-cache-config.xml"
DEBUG - 2010-07-23 23:43:09.515 <D4> (thread=main, member=n/a): TCMP bound to /192.168.1.10:7070 using SystemSocketProvider
INFO - 2010-07-23 23:43:41.000 <Info> (thread=Cluster, member=n/a): Created a new cluster "WORKSTATION" with Member(Id=1, Timestamp=2010-07-23 23:43:09.531, Address=192.168.1.10:7070, MachineId=26890, Location=site:local,machine:WS,process:5880, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=1) UID=0xC0A8010A0000012A01427FDB690A1B9E
INFO - 2010-07-23 23:43:41.015 <Info> (thread=main, member=n/a): Started cluster Name=WORKSTATION
WellKnownAddressList(Size=1,
WKA{Address=192.168.1.10, Port=7070}
MasterMemberSet
ThisMember=Member(Id=1, Timestamp=2010-07-23 23:43:09.531, Address=192.168.1.10:7070, MachineId=26890, Location=site:local,machine:WS,process:5880, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2010-07-23 23:43:09.531, Address=192.168.1.10:7070, MachineId=26890, Location=site:local,machine:WS,process:5880, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2010-07-23 23:43:09.531, Address=192.168.1.10:7070, MachineId=26890, Location=site:local,machine:WS,process:5880, Role=CoherenceServer)
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0, BitSetCount=0
TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}
DEBUG - 2010-07-23 23:43:41.093 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
DEBUG - 2010-07-23 23:43:41.718 <D5> (thread=main, member=1): parametrized policy class instance created
DEBUG - 2010-07-23 23:43:41.718 <D5> (thread=main, member=1): p_info=jdbc-coherence/MY-CACHE-DS
DEBUG - 2010-07-23 23:43:41.750 <D5> (thread=DistributedCache:test-guardable-service, member=1): Service test-guardable-service joined the cluster with senior service member 1
INFO - 2010-07-23 23:43:41.890 <Info> (thread=main, member=1):
Services
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.6, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
PartitionedCache{Name=test-guardable-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
Started DefaultCacheServer...
DEBUG - 2010-07-23 23:44:50.250 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2010-07-23 23:44:50.258, Address=192.168.1.10:7072, MachineId=26890, Location=site:local,machine:WS,process:3376, Role=TestGuardianTestGuardianCache) joined Cluster with senior member 1
DEBUG - 2010-07-23 23:44:50.390 <D5> (thread=Cluster, member=1): Member 2 joined Service Management with senior member 1
DEBUG - 2010-07-23 23:44:51.093 <D5> (thread=Cluster, member=1): Member 2 joined Service test-guardable-service with senior member 1
DEBUG - 2010-07-23 23:44:51.218 <D5> (thread=DistributedCache:test-guardable-service, member=1): inside cacheloader constructor, p_datasource_name=jdbc-coherence/MY-CACHE-DS
DEBUG - 2010-07-23 23:44:51.359 <D5> (thread=test-guardable-serviceWorker:4, member=1): inside cacheloader
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=Recovery Thread, member=1): inside onGuardableRecovery
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=Recovery Thread, member=1): p_guardable=Guard{Daemon=test-guardable-serviceWorker:4}
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=Recovery Thread, member=1): p_guardable.getClass().getCanonicalName()=com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Daemon$Guard
ERROR - 2010-07-23 23:45:00.375 <Error> (thread=DistributedCache:test-guardable-service, member=1): Attempting recovery (due to soft timeout) of {WrapperGuardable Guard{Daemon=test-guardable-serviceWorker:4} Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}}
WARN - 2010-07-23 23:45:00.375 <Warning> (thread=Recovery Thread, member=1): A worker thread has been executing task: Message "GetRequest"
FromMember=Member(Id=2, Timestamp=2010-07-23 23:44:50.258, Address=192.168.1.10:7072, MachineId=26890, Location=site:local,machine:WS,process:3376, Role=TestGuardianTestGuardianCache)
FromMessageId=29
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=59
ToPollId=0
Poll=null
Packets
[000]=Directed{PacketType=0x0DDF00D5, ToId=1, FromId=2, Direction=Incoming, ReceivedMillis=23:44:51.343, ToMemberSet=null, ServiceId=2, MessageType=59, FromMessageId=29, ToMessageId=27, MessagePartCount=1, MessagePartIndex=0, NackInProgress=false, ResendScheduled=none, Timeout=none, PendingResendSkips=0, DeliveryState=unsent, Body=0}
Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
ToMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2010-07-23 23:43:09.531, Address=192.168.1.10:7070, MachineId=26890, Location=site:local,machine:WS,process:5880, Role=CoherenceServer)
NotifySent=false
} for 9016ms and appears to be stuck; attempting to interrupt: test-guardable-serviceWorker:4
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=test-guardable-serviceWorker:4, member=1): loader thread was interrupted
at java.lang.Thread.sleep(Native Method)
at test.guardian.testCacheStore.load(testCacheStore.java:26)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.load(ReadWriteBackingMap.java:4759)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.loadInternal(ReadWriteBackingMap.java:4344)
at com.tangosol.net.cache.ReadWriteBackingMap.get(ReadWriteBackingMap.java:807)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onGetRequest(PartitionedCache.CDB:22)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$GetRequest.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:595)
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=test-guardable-serviceWorker:4, member=1): loader stop_thread flag was set to stop thread
DEBUG - 2010-07-23 23:45:00.375 <D5> (thread=test-guardable-serviceWorker:4, member=1): return value from the store
DEBUG - 2010-07-23 23:45:00.390 <D5> (thread=Cluster, member=1): TcpRing disconnected from Member(Id=2, Timestamp=2010-07-23 23:44:50.258, Address=192.168.1.10:7072, MachineId=26890, Location=site:local,machine:WS,process:3376, Role=TestGuardianTestGuardianCache) due to a peer departure; removing the member.
DEBUG - 2010-07-23 23:45:00.390 <D5> (thread=Cluster, member=1): Member 2 left service Management with senior member 1
DEBUG - 2010-07-23 23:45:00.390 <D5> (thread=Cluster, member=1): Member 2 left service test-guardable-service with senior member 1
DEBUG - 2010-07-23 23:45:00.390 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2010-07-23 23:45:00.39, Address=192.168.1.10:7072, MachineId=26890, Location=site:local,machine:WS,process:3376, Role=TestGuardianTestGuardianCache) left Cluster with senior member 1
ERROR - 2010-07-23 23:48:16.250 <Error> (thread=Cluster, member=1): Attempting recovery (due to soft timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:test-guardable-service} Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}}
DEBUG - 2010-07-23 23:48:16.250 <D5> (thread=Recovery Thread, member=1): inside onGuardableRecovery
DEBUG - 2010-07-23 23:48:16.250 <D5> (thread=Recovery Thread, member=1): p_guardable=Guard{Daemon=DistributedCache:test-guardable-service}
DEBUG - 2010-07-23 23:48:16.250 <D5> (thread=Recovery Thread, member=1): p_guardable.getClass().getCanonicalName()=com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard
DEBUG - Interrupted PartitionedCache, Thread[DistributedCache:test-guardable-service,5,Cluster]
ERROR - 2010-07-23 23:48:46.765 <Error> (thread=Cluster, member=1): Terminating guarded execution (due to hard timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:test-guardable-service} Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}}
DEBUG - 2010-07-23 23:48:46.765 <D5> (thread=Termination Thread, member=1): inside onGuardableTerminate
DEBUG - 2010-07-23 23:48:46.765 <D5> (thread=Termination Thread, member=1): p_guardable=Guard{Daemon=DistributedCache:test-guardable-service}
DEBUG - 2010-07-23 23:48:46.765 <D5> (thread=Termination Thread, member=1): p_guardable.getClass().getCanonicalName()=com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard
DEBUG - 2010-07-23 23:48:46.765 <D5> (thread=DistributedCache:test-guardable-service, member=1): Service test-guardable-service left the cluster
INFO - 2010-07-23 23:48:46.890 <Info> (thread=main, member=1): Restarting Service: test-guardable-service
DEBUG - 2010-07-23 23:48:46.906 <D5> (thread=main, member=1): parametrized policy class instance created
DEBUG - 2010-07-23 23:48:46.906 <D5> (thread=main, member=1): p_info=jdbc-coherence/MY-CACHE-DS
DEBUG - 2010-07-23 23:48:46.906 <D5> (thread=DistributedCache:test-guardable-service, member=1): Service test-guardable-service joined the cluster with senior service member 1
ERROR - 2010-07-23 23:48:55.265 <Error> (thread=Cluster, member=1): Attempting recovery (due to soft timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:test-guardable-service} Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STOPPED), Not initialized}}
DEBUG - 2010-07-23 23:48:55.265 <D5> (thread=Recovery Thread, member=1): inside onGuardableRecovery
DEBUG - 2010-07-23 23:48:55.265 <D5> (thread=Recovery Thread, member=1): p_guardable=Guard{Daemon=DistributedCache:test-guardable-service}
DEBUG - 2010-07-23 23:48:55.265 <D5> (thread=Recovery Thread, member=1): p_guardable.getClass().getCanonicalName()=com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard
ERROR - 2010-07-23 23:48:56.281 <Error> (thread=Cluster, member=1): Terminating guarded execution (due to hard timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:test-guardable-service} Service=PartitionedCache{Name=test-guardable-service, State=(SERVICE_STOPPED), Not initialized}}
DEBUG - 2010-07-23 23:48:56.281 <D5> (thread=Termination Thread, member=1): inside onGuardableTerminate
DEBUG - 2010-07-23 23:48:56.281 <D5> (thread=Termination Thread, member=1): p_guardable=Guard{Daemon=DistributedCache:test-guardable-service}
DEBUG - 2010-07-23 23:48:56.281 <D5> (thread=Termination Thread, member=1): p_guardable.getClass().getCanonicalName()=com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$Guard -
How to limit Write-Behind batch
We have a scenario:
we use read-write-backing-map-scheme having write-delay 60s.
System insert a lot of data and then time comes to write data coherence find 40-50 k of unsaved record and pass them all to cachestore.
Due to data volume or database busyness cashstore may work some time several seconds for instance.
<read-write-backing-map-scheme>
<scheme-name>TicketDatabaseScheme</scheme-name>
<scheme-ref>DefaultDatabaseScheme</scheme-ref>
<!--<write-delay>1m</write-delay>-->
<cachestore-scheme>
<class-scheme>
<class-name>com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore</class-name>
</class-scheme>
</cachestore-scheme>
</read-write-backing-map-scheme>
<read-write-backing-map-scheme>
<scheme-name>DefaultDatabaseScheme</scheme-name>
<internal-cache-scheme>
<local-scheme>
<scheme-ref>LocalScheme</scheme-ref>
</local-scheme>
</internal-cache-scheme>
<write-delay>60s</write-delay>
</read-write-backing-map-scheme>
In the case we experience "Terminating guarded execution" followed by service termination.
2010-02-11 09:26:52.223/511.457 Oracle Coherence GE 3.5.2/463 <Error> (thread=DistributedCache:TicketonCache, member=2): Terminating guarded execution (due to hard timeout 1924ms ago) of Daemon{Thread="Thread[WriteBehindThread:CacheStoreWrapper(com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore),5,WriteBehindThread:CacheStoreWrapper(com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore)]", State=Running}
2010-02-11 09:26:52.225/511.459 Oracle Coherence GE 3.5.2/463 <Error> (thread=Termination Thread, member=2): Write-behind thread timed out; stopping the cache service
2010-02-11 09:26:52.226/511.460 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache:TicketonCache, member=2): Service TicketonCache left the cluster
INFO 09:26:52,227 [http--80-22$27432016 DaoCoherenceImpl] - PROFILE_doCreatetickets putAll 200 tickets time 3444 time per 10 objects 172
INFO 09:26:52,227 [http--80-22$27432016 DaoCoherenceImpl] - PROFILE event and 1000 tickets ctreated time 9668
Broadcast Message from root (msglog) on ip-10-226-137-172 Thu Feb 11 09:57:18...ets putAll 200 tickets time 3365 time per 10 objects 168
2010-02-11 09:26:52.228/511.462 Oracle Coherence GE 3.5.2/463 <Info> (thread=httTHE SYSTEM ip-10-226-137-172 IS BEING SHUT DOWN NOW ! ! !et
Log off now or risk your files being damagedence GE 3.5.2/463 <Info> (thread=http--80-27$25787595, member=2): Restarting Service: TicketonCache
INFO 09:26:52,229 [http--80-20$15974570 DaoCoherenceImpl] - PROFILE_doCreatetickets putAll 200 tickets time 3447 time per 10 objects 172
INFO 09:26:52,289 [http--80-22$26935588 BackingBeanSuper] - request HttpRequest[22]
[09:26:53.446] {http--80-35$24027494} java.lang.RuntimeException: Failed to start Service "TicketonCache" (ServiceState=SERVICE_STOPPED)
[09:26:53.446] {http--80-35$24027494} at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
Resume: coherence crashes in case far from fatal.
Questions
1. May i limit the size of a batch passed to cachestore?
2. Is it possible to configure the timeout "(due to hard timeout 1924ms ago)"
3. Is it possible to handle like this some way to prevent self killing of coherence cluster.Thank you Mark you are vary helpfull
Did you mean that (lower) by "bundle strategy"?
<cachestore-scheme>
<class-scheme>
<class-name>com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore</class-name>
</class-scheme>
<operation-bundling>
<bundle-config>
<operation-name>store</operation-name>
<preferred-size>5000</preferred-size>
<auto-adjust>true</auto-adjust>
</bundle-config>
</operation-bundling>
</cachestore-scheme>
And if yes is it looks sense?
I mean by this, "send records to TicketCacheStore by 5000 per call " am i right?
I dropped delay to 10s and set factor to 0.5
Not coherece send me 5-20k records and cachestore handle whis successfuly.
But! By diferent means it may work longer sometimes, some lock in database for instance.
I want to find durable solution for the case, not only lower a chance i meet one.
Issuing heartbeat from cachestore looks best for me now.
I find that default guardian timeout is 65s and it is not looks as good idea to make it higher.
Maybe you are looking for
-
Lost mask option button for adjustment brush
I am not well up on the correct lingo so bare with me please. When using adjustment brush and Crop tool a bar used to appear under the photo giving me the option of seeing the overlay filter of where i had burned in or dodged. it also appeared for th
-
Hmmm - what a mess. I downloaded ML - took many hours, then in the morning clicked "install". It went through it's gyrations, but I noticed that my screen kept toggling to "analog powersaving mode" and shutting off. Now that it is fully installed,
-
Can anyone explain how to create this effect in Deamweaver and Fireworks?
This site was oriinally done in dreamweaver - but the shadow effect was done outside of Dreamweaver. When I save to html and load into Dreamweaver it makes no sense to me. Can anyone explain how to create this effect in Deamweaver and Fireworks? http
-
How to generate two files in DME
Hi, I copied standard program RFFOBR_U and created customized program for my bank. Now I need to generate 2 files in DME based on vendors of the bank. How to do this? Ezhil.
-
What is the best way to handle canonical links?
I am migrating a client website over to BC and I've noticed they have canonical links in most, if not all of the HEAD sections for each page. How should I go about maintaining these canonical links in BC .... since the HEAD section resides in the tem