Submit submit a large number of task to a thread pool (more than 10,000)

Question

i want to submit a large number of task to a thread pool (more than 10,000).
Since a thread pool take runnable as input i have to create as many objects of Runnable as the number of task, but since the number of task is very large it causes the memory overflow and my application crashes.
Can you suggest me some way to overcome this problem?

Answer 1

Ravi_Gupta wrote:
I have to serve them infinitely depending upon the choice of the user.
Take a look at my code (code of MyCustomRunnable is already posted)
public void start(Vector<String> addresses)
searching = true;What is this for? Is it a kind of comment?
>
Vector<MyCustomRunnable> runnables = new Vector<MyCustomRunnable>(1,1);
for (String address : addresses)
try
runnables.addElement(new MyCustomRunnable(address));
catch (IOException ex)
ex.printStackTrace();
}Why does MyCustomRunnable throw an IOException? Why is using up resources when it hasn't started. Why build this vector at all?
>
//ThreadPoolExecutor pool = new ThreadPoolExecutor(100,100,50000L,TimeUnit.MILLISECONDS,new LinkedBlockingQueue());
ExecutorService pool = Executors.newFixedThreadPool(100);You have 100 CPUs wow! I can only assume your operations are blocking on a Socket connection most of the time.
>
boolean interrupted = false;
Vector<Future<String>> futures = new Vector<Future<String>>(1,1);You don't save much by reusing your vector here.
for(int i=1; !interrupted; i++)You are looping here until the thread is interrupted, why are you doing this? Are you trying to generate loading on a remote server?
System.out.println("Cycle: " + i);
for(MyCustomRunnable runnable : runnables)Change the name of you Runnable as it clearly does much more than that. Typically a Runnable is executed once and does not create resources in its constructor nor have a cleanup method.
futures.addElement((Future<String>) pool.submit(runnable));Again, it unclear why you would use a vector rather than a list here.
>
for(Future<String> future : futures)
try
future.get();
catch (InterruptedException ex)
interrupted = true;If you want this to break the loop put the try/catch outside the loop.
ex.printStackTrace();
catch (ExecutionException ex)
ex.printStackTrace();If you are generating a load test you may want to record this kind of failure. e.g. count them.
futures.clear();
try
Thread.sleep(60000);Why do you sleep even if you have been interrupted? For better timing, you should sleep, before check if you futures have finished.
catch(InterruptedException e)
searching = false;again does nothing.
System.out.println("Thread pool terminated..................");
//return;remove this comment. its dangerous.
break;why do you have two way of breaking the loop. why not interrupted = true here.
searching = false;
System.out.println("Shut downing pool");
pool.shutdownNow();
try
for(MyCustomRunnable runnable : runnables)
runnable.close(); //release resources associated with it.
catch(IOException e)put the try/catch inside the loop. You may want to ignore the exception but if one fails, the rest of the resources won't get cleaned up.
The above code serve the task infinitely untill it is terminated by user.
i had created a large number of runnables and future objects and they remain in memory until
user terminates the operation might be the cause of the memory overflow.It could be the size of the resources each runnable holds. Have you tried increasing your maximum memory? e.g. -Xmx512m