Multithreading application - how to use correctly if uses network?

Question

I'm coding "daemon"-like software, which do some things in the loop (login to somewhere, do some things and logout) using multithreading.
My application run about 1000-2000 threads and this threads do job.
So here is my htop log at first ~10 minutes:
http://i.stack.imgur.com/wlaeh.png
So you can see on the above screenshot that some number of threads is running (ordered by green R status)
But after some time this happens:
http://i.stack.imgur.com/xf9Ny.png
As you can see just one-three threads are running periodically (it seems like system slow down threads or make it sleeping).
Application run threads for small-time job (it's about 10 seconds to 1 minute) and after thread die Fixed Thread Pool Executor create new threads to do same job.
So as I understand it should be equal speed of threads execution at all times. But it slow down after ~10-20 minutes. Normally running threads in htop up to ~500-200-300-100 threads and down to ~3-30-60 threads at each time. But after this 10-20 minutes just only thread is executing. So can you tell me what is happens and how to fix that?
Or can please anybody contact me at email *[REMOVED BY MODERATOR]* and I'll give source code (it's not big) and *[REMOVED BY MODERATOR]* ?
Edited by: 907212 on Jan 10, 2012 5:45 PM
Edited by: EJP on 11/01/2012 14:57: moderated. Don't post your email address in public places; and this is not a rent-a-coder service. Answers should be posted here.

Answer 1

You don't mention what hardware you are runnng this on and what your software configuration is (os version, java version).
But unless your are on a really big system you will never get 1000 - to 2000 threads doing anything at the same time.
The number of threads that can execute simultaneously is mostly governed by hardware related to the number of processors and the number of cores.
And if a thread isn't one of those that is executing simultaneously then it must be sleeping and not doing any work. With that many threads the biggest percentage of them will always be sleeping.
Threads that are executing will keep executing until they relinquish control (to wait for some event) to another thread; unless you have implemented a homegrown time-slice mechanism that doles out fixed-interval slices to threads in a round-robin fashion.
What drove the decision to use so many threads?
We can provide some suggestions if you give us some information about what the business requirement actually is for your application. Include information about what type of work the threads will actually be doing.
You included one example of logging in, doing some work, and logging out. How many different systems are you connecting to? Are they all Oracle DBs? How often do you connect/reconnect to the same Oracle DB? Depending on the use case your app could benefit by using a connection pool that keeps connections open; there is a high performance cost for opening/closing/reopening connections.