Do more Number of Threads reduce task duration

Hi,
Ihave made an application which 100 parse an XML and writes it to a different file. And it takes some time this task is done is different thread. If I use 5 threads or let say 10 threads to perform this task. Will it make any difference or it will take some what same amount of time.
Thanks & Regards
Ritesh

It all depends on how fast your discs are etc. I think
the slow part is to read/write from/to disc, and too
many threas will only make it worse (since the disc
has to read/write from several places on the disc).
Not just that but the scheduling of those multiple threads eat up CPU time (and a bit of RAM) as well.
And unless you have a multi-CPU machine they won't run truly in parallel.
So it could well take longer to complete the entire task (even if small files might appear faster if processed in different threads from long one, those long ones will then take longer).

Similar Messages

Will more number of waiting threads in thread pool degrade performance?

Hi,
I use thread pool and set the maximum number of threads to execute. My problem is, the thread pool initiates the number of threads and these threads wait until they get a work. Will this degrade the performance (The threads do I/O operations) ??

Threads waiting for work will not degrade performance. If your work involves those threads waiting for I/O then that in itself will not degrade performance either (as long as they block and don't poll of course).
All live threads consume resources however so if you are short on threads or memory then don't create too many of them.
Pre-starting a large number of threads in a pool can cause a startup delay of course. Generally you should let the pool create threads as needed until it gets to the core pool size.

High number of threads and high CPU usage on a single instance

Without any apparent reason, in some moments of day the website reports an increment of the HTTP Queue Length from an average of 10/20 queued requests to 100/200 and more.
In that period of time the Average Response Time increases but the number of the requests on the website don't. Even increasing the instances it doesn't help to resolve the problem (maybe it just reduced it).
It seems that the problem is not related to the traffic on the site. This issue happens either when the traffic is high or low.
The autoscaling works well, we don't have peak of traffic but always a slow increasing of requests and the CPU is always below the 50% (memory too). The issue is resolved by the swap of the site and sometimes it's resolve by itself after a while.
What I discovered when the issue happens, is that there's an instance (just an instance only), that has an high number of threads and CPU usage on the W3WP.exe process and these values are above the mean than the other instances. Eg, instances are around
50/60 threads and 10/20% of CPU, the instance with the problem has 200 threads and 50/60% of CPU. If I kill the W3WP process on that instance the issue is resolved.
I monitored the HTTP requests, I tried the website extension "Diagnostics as service" but I can't discover anything that can help me to understand the problem. Of course, there are many requests that go in timeout but it is a consequences of a
unresponsive instance.
On the web hosting plan there are only two sites: the production site and its staging which is used for update of the production only and it's always sleeping. Plus, no webjobs are running on this site.
What can I do to gather useful information that can help me to understand the reason of this problem?
Thank you!

"Does the instance with the high counts receive more traffic than the other instances? Is it possible that the load balancer not
working the way it should be?"
How can I get that information? I can't see metrics for a specific instance
"Does it always happen to the first instance?"
I will check it better, but in the order given by the Processes panel it's the second instance which has the issue. The scaling is at 2 instances for the most part of the time (it's the minimum).
Maybe one time it was the 3th instance but I'm not sure, I'll give it more attention.
" How long do these moment last?"
The time can be 10 to 30 minutes, but I fix it as soon as I see the problem. It's not the down-scaling to resolve it because in these situations the CPU as well is high so the scaling holds or increases the number of instances.
"- How often do these moments occurs?"
It occurs quite often, 2/4 times a week.

Max number of threads in Java?

Hi,
I am running into a bug now that I am testing my working code. Basically, my program creates a bunch of objects that talk to each other. Each object is a thread, and at a given time (there is a one object that keeps track of time) a thread may choose to message another thread. My problem is that when I create more objects, my code doesn't run to completion. For example, when I have 10 objects talking to each other, things work fine. But if I have 15, they all run for a little bit, then the program seems to just hang and not do anything.
Is there a limit as to how many threads I can have? If I was out of memory or something, I'd get an exception, right? But I don't get anything - it just seems like things "freeze up".

15 threads? shouldn't be a problem.
I think th emax threads allows runs in thousands..if not ten of thousands.Java process threads itself use about 600 or more threads. howevr, it could also be the underlying system restr8iction on the number of threads allow per process????
for you case: it's highly likely you run into the max # of threads.
I can only think of two reasons for your application to freeze:
1. Deadlock
if you have an object that is synchronized..than you might have run into a deadlock (race-condition)
for example:
Object A has the key for Object X , and waiting for Object Z
Object B has the key for Object Z, and waiting for Object X
as you can see..they will never gives up the lock....so you're in a dead lock.
2. Low memory resource. (Memory - Paging and Thrashing)
Each of your thread is using up the resouces (memory) and processing power.
When you reach the max or near max and needs to create more memory..the garbage collector kick in
and try to reclaim some unused memory. This can slow down your application dramatically if the garbage
collector is invoke often.
Also..paging is performed when you reach max memory..the operating system keep on paging your memory (usually happens when there's a lot of threads and not enough memory. If this happen..than it
can cause your program to becomes freeze like....remeber..each thread is given a small amount of time to
perform a task..if the time it takes to load a page for a thread is almost equals to context switch time..than
no work is really done..and your program "freeze"
solution..redesign you app to prevent thrashing.
it is likely paging is the culprit..but i would not dismiss deadlock issues.

Limiting the number of threads that are created

I am trying to write a web crawler as a way of learning about multi-threading and I have hit a stumbling block (probably my design). The way the application I have started works is that I provide a single URL, which is passed to a thread and that thread then parses the web page building a list of links (If the links are not already held in a 'visited' list then they are saved in a list in the Runnable object that is parsing the html page).
When the page has been parsed the list of links are then passed to a method in a Utilities class that creates a new group of threads, 1 for each link and each of these threads then does the same thing. The problem I have is that as more and more links are captured more and more threads are created eventually I either get a out of memory exception or I get an operating system 'cannot create native thread' message. I am not sure if the threads are staying in memory after they have done their tasks, I am not sure why I am running out of memory so quickly.
What I would like to try and do is to set a limit for the maximum number of threads created and then only create new threads if the limit is not breached. If someone could point me in the right direction that would be good, I have googled around but cant find an example of what I want to do.
Thanks

Thanks for that, that has given me a lot to read up on and I can already see where I went wrong.... I think implementing a thread pool and also a work queue is probably the way I will go from now on.

Maximum number of threads

I have an application for which I am interested in obtaining as many simultaneous
threads as possble (e.g. 10's and possibly 100's of thousands of threads).
More specifically I have developed a simulation language formalism in which the
dynamics of each "entity" moving through a network of queues is coordinated by
a separate thread of execution.
This is not a traditional application where all or even some small number are
intended to run concurrently. A single executive thread is responsible for alternately
activating and deactivating individual threads to reflect the time ordered sequencing
of events corresponding to each entity's movement through the system of queues.
As a result, only a single thread is executing at one time.
Using a separate thread for each entity off-loads the task of managing each's
execution context to the JVM. The resource contention issues that arise in a
standard multi-threaded application don't apply here.
For "large" systems though (ones populated by many entities simultaneously) I
need many threads.
I'm using JRockit's thin thread model on a Windows XP machine and am "only" able
to get approximately 25000 threads at which point the JVM hangs with no diagnostic.
I say "only" because at that point there seems to be plenty of heap space left
as well as Page File space left in the XP's Virtual Memory system.
I'm using the following JVM command line:
java -Xthinthreads -Xss32K -Xms256M -Xmx256M -Xallocationtype:global
Surprisingly, by altering the heap size upward from 256MB to 512MB acutally causes
the VM to hang with fewer threads (only 20000 or so)
Using allicationtype:global and altering the stack size of the threads seems to
have little effect on the maximum number attainable.
It seems the maximum number of threads attaible should be a function of the size
of the heap space, and the amount of physical and virtual memory limits on the
system.
My empirical evidence, however, seems to indicate there is some internal VM limit.
Can someone explain for me the implementation of the thin thread model and its
limitations w.r.t. the number of threads it can support?
Thanks in advance for you help.

You might try to use the new java.util.concurrent.* api available in j2se
1.5.0 as
a base for your implementation instead.
/Robert
"Staffan Larsen" <[email protected]> wrote in message
news:40f63c6b$1@mktnews1...
The reason for the "artifical" limitation of 32767 is that each thread
needs to have it's own id. The way the object synchonization algorithm
works, there has to be space for this id in the header of each object.
The header has a very limited amount of space (currently 2 words) and
thus there is not space for a larger thread id than 32767.
You may be interested (but sad) to know that the thinthreads model has
been removed from later versions of JRockit. The reason was the overhead
of maintaining and supporting multiple threading systems was just too
much work for a small return. Although, for very specialized systems
(like yours) it would give a higher return.
About your calculation. You said in you first post that you set -Xss32k
which means that each thread will take at least 32k memory, more likey
40k with other overhead. That means about 25 threads per MB or 25000 per
GB, given that everything scales linearly.
Regards,
/Staffan
Kevin Healy wrote:
Staffan,
Thanks for your response. Certainly in a traditional multi-threaded
applications
where there exist many runnable threads competing for resources, thepoint of
diminishing returns is usually on the order of 10's of threads; but, asI mentioned,
this is not a traditional application. In this case, the threadingframework
is used as a natural and convenient way to manage the execution contextof the
(many) entities in the system. Since I'm not interested in trueconcurrency or
exploiting parallelism, I view threads as just data and it strikes methat I should
be able to get as many as I have room for. In fact the green threadsmodel of
the pre JDK 1.4 JVM's was well-suited to this kind of application but itseems
the JVM implementors have not considered this in removing support forgreen threads
in their newest releases. The thin threads model of JRockit is the nextbest
thing but the 32767 limit seems entirely artificial.
Do you know of any way to bump up that limitation or who I might contactat BEA
about the matter?
I'm aware that stack space and other context specific data must beallocated for
each thread so let's say for the sake of argument that each thread takes10KB
of data. That means I get 100 threads per 1MB and it would take 1GB toget my
100,000 threads. That is not an unusually large amount of data ontoday's desktop
computer. Furthermore, with the advent of 64bit computing on thedesktop, we'll
see machines supporting much more than 4GB.
Staffan Larsen <[email protected]> wrote:
First: which version of JRockit is this with?
With thinthreads the limit is 32767 threads. But you may run into other
limits before that. When you increase the heapsize you "steal" memory
from the system (which would otherwise be used for stacks and other data
structures) and that is why you can run fewer threads with a larger
heap.
>>>
I don't think it is realistic to run 100's of thousands of threads in
one single system.
Regards,
/Staffan
Kevin Healy wrote:
I have an application for which I am interested in obtaining as manysimultaneous
threads as possble (e.g. 10's and possibly 100's of thousands of
threads).
>>>>
More specifically I have developed a simulation language formalismin which the
dynamics of each "entity" moving through a network of queues is
coordinated
>>>
by
a separate thread of execution.
This is not a traditional application where all or even some smallnumber are
intended to run concurrently. A single executive thread is responsiblefor alternately
activating and deactivating individual threads to reflect the timeordered sequencing
of events corresponding to each entity's movement through the systemof queues.
As a result, only a single thread is executing at one time.
Using a separate thread for each entity off-loads the task of managingeach's
execution context to the JVM. The resource contention issues thatarise in a
standard multi-threaded application don't apply here.
For "large" systems though (ones populated by many entities
simultaneously)
>>>
I
need many threads.
I'm using JRockit's thin thread model on a Windows XP machine and am"only" able
to get approximately 25000 threads at which point the JVM hangs withno diagnostic.
I say "only" because at that point there seems to be plenty of heapspace left
as well as Page File space left in the XP's Virtual Memory system.
I'm using the following JVM command line:
java -Xthinthreads -Xss32K -Xms256M -Xmx256M -Xallocationtype:global
Surprisingly, by altering the heap size upward from 256MB to 512MBacutally causes
the VM to hang with fewer threads (only 20000 or so)
Using allicationtype:global and altering the stack size of the threadsseems to
have little effect on the maximum number attainable.
It seems the maximum number of threads attaible should be a functionof the size
of the heap space, and the amount of physical and virtual memory limitson the
system.
My empirical evidence, however, seems to indicate there is some
internal
>>>
VM limit.
Can someone explain for me the implementation of the thin thread modeland its
limitations w.r.t. the number of threads it can support?
Thanks in advance for you help.

Grey screen and "incorrect number of thread records"

I think I am in trouble. My eMac will not boot and gets stuck on the great screen with the Apple logo and the spinning windmill. I have followed the instructions in the Apple help article up to but not including "archive and install". I tried the advanced secure boot and the messages I got over and over were "HFSSWAPHFSPlusBTIternalmode: catalog record #30 keylength=34 expected=144 node=15278 File ID=4 volume= Macintosh HD device=/dev/disk 0s 10"
And
"Jan31 12:00:35 Launched: /System /Library/CoreServices/Loginwindow.app/Contents/MacOS/Loginwindow Port/dev/console exited abnormally: Trace/ BPT trap"
And
Jan31 12:02:13 Launched: Getty repeating too quickly on port/ dev/ console, sleeping"
I rest the PRAM and NVRAm as instructed.
I booted from the OSX 10.4 install disk and ran the verify disk and repair disk sequences several times. It did repair a number of things but it kept coming back to "
Message was edited by: Mont Rigaud Stud
Itkept coming back to "incorrect number of thread records". Even after running repair disk again several times it came back to the same thing and said
"the volume Macintosh HD could not br repaired after 3 attempts
ERROR: the underlying task reported failure On exit
1HFS volume checked
1 vOlume could not be repaired because of an error
Repair attempted on 1 volume
1 volume could not be repaired"
What should I do? I am NOT BACKED UP! How do I back up now? Can I usean external Hard drive? I really do not want to lose my photOs, email and business files.
I am typing my question on my iPhone, so that accounts for the poor capitalization.
Thanks
Doug

Will it boot in Safe Mode? See
What is Safe Boot, Safe Mode? (Mac OS X)
http://docs.info.apple.com/article.html?artnum=107392
Takes a while to run, but it usually "fixes" problems.
Resolve startup issues and perform disk maintenance with Disk Utility and fsck
http://docs.info.apple.com/article.html?artnum=106214
 Cheers, Tom

Importing From Excel file will not import assignments correctly where there are more than one assignment per task

I am trying to import an excel file ... see below. It appears if you select both Tasks and Assignments tables it will not allow you to do this, says file is not in a project recognized format. So I import Tasks tab first then run the
wizard again and import the assignments tab while selecting merge with project option. This works with one exception:
If there are more than one assignment per task. It does not merge in the 2nd assignment, only the first.
Is there a way to import a task with 2 or more assignments?
Background
To figure out what the format for the file needed to be, I started with a MPP file and from project saved as Excel and selected the "Project Template" file. Now I am trying to import that XLS to see how it works
ID
Active
Task Mode
Name
Duration
Start
Finish
Predecessors
Outline Level
1
Yes
Auto Scheduled
DS001
3 days
June 17, 2014 8:00 AM
June 19, 2014 5:00 PM
1
2
Yes
Auto Scheduled
MT001
3 days
June 17, 2014 8:00 AM
June 19, 2014 5:00 PM
2
3
Yes
Auto Scheduled
CT001
1 day
June 17, 2014 8:00 AM
June 17, 2014 5:00 PM
3
4
Yes
Auto Scheduled
CT002
2 days
June 18, 2014 8:00 AM
June 19, 2014 5:00 PM
3
3
5
Yes
Auto Scheduled
DS002
1 day
June 20, 2014 8:00 AM
June 20, 2014 5:00 PM
1
6
Yes
Auto Scheduled
MT002
1 day
June 20, 2014 8:00 AM
June 20, 2014 5:00 PM
2
7
Yes
Auto Scheduled
CT003
1 day
June 20, 2014 8:00 AM
June 20, 2014 5:00 PM
4
3
Task Name
Resource Name
% Work Complete
Work
Units
CT001
Engineer1
0
8h
100%
CT002
Engineer2
0
16h
100%
CT003
Engineer1
0
8h
100%
CT003
Engineer2
0
8h
100%
Andrew Payze

Andrew,
I did a quick test using your example although I didn't include the Predecessor or Outline Level fields to keep things a little simpler. I imported the Excel data into a new Project file with a single import of task and assignment data. It is very important
to set up the Excel Workbook correctly. On Sheet 1 I entered the task data (Name, Duration, Start). I did not include the ID or finish date as that is redundant - Project generates its own ID and will calculate the finish date based on the start date and duration.
On Sheet 2 I entered the assignment data basically just as you show it except I didn't include the Units.
The data imported as expected.
John

"max number of thread"

We are running a java web app on a Ubuntu Oracle WebLogic server version 10.3.3.
The java web app performs long polls with open tcp socket to keep the client connection open. The clients are long polling the WebLogic at 30 seconds.
Currently we are not able to maintain stability for greater than 24 hours with approximately 200 simultaneous sessions on WebLogic server. Session to me is active client/server tcp connection. We have re-written our application to use continuations, but we are seeing ConcurrentModificationException errors in performance testing.
Is there any setting in WebLogic for “max number of thread” that can handle?
Edited by: user9316392 on Jul 8, 2010 11:07 AM

First, WebLogic since 9.0 has a self-tuning thread pool where WLS will automatically grow and shrink the number of threads based on some internal algorithms. I'm not aware of a hard limit so theoretically there is no max thread count as long as the JVM has memory and WLS thinks more threads will help. You can read up on it here:
http://www.oracle.com/technology/pub/articles/dev2arch/2006/01/workload-management.html
Practically, I wouldn't expect more than several hundred threads to be helpful.
As for your situation, how does WLS become unstable? Out of memory, out of file descriptors, errors on new requests, etc. I think you're going to have to use some JVM tools to see what happens to your JVM over time. Is there a memory leak somewhere, is it non-heap memory, etc. JRockit Mission Control is helpful if running on JRockit. If you're on Sun Hotspot, them presumably you can use some of the Hotspot tools. You'll want to compare the state of the JVM towards the beginning of your load test, but after a slight warm-up period with a snapshot after the load test has been running for a long period of time.

Max number of threads in a jvm

Is there any constraint for the maximum number of threads that we can
create in a particular JVM

Why do you have to worry about that? I don't see a reason to create lots and lots of threads. The OS will usually spend most of the time doing context switches if you create more than a couple of thousand threads.
Kaj

Error 200524 Different number of channels in task and data

Anyone out there with some information would be greatly appreciated. I have attached my VI. Essentially I am trying to build a simple PID VI to controll the temperature of a piece of equipment in a system I am building. I have heating wire powered by 120 V source. The circuit is controlled by a simple relay, which is controlled by a digital line on my NI USB 6343 board. I have seen numerous examples throughout the forums and the shipped examples, but I cannot seem to figure out how the PID VIs work. I understand the basics of a PID controller, but cannot make the connections of how to actually get it to work in LV.
I figured that this VI I wrote is pretty simple and should work, but I keep getting this error.
Error -200524 occurred at PID_v0.vi:Instance:19:1
Possible reason(s):
Write cannot be performed, because the number of channels in the data does not match the number of channels in the task.
When writing, supply data for all channels in the task. Alternatively, modify the task to contain the same number of channels as the data written.
Number of Channels in Task: 1
Number of Channels in Data&colon; 1000
Task Name: _unnamedTask<1>"
Any advice, direction, sample VIs that are not overly complicated, etc would be greatly appreciated.
Attachments:
PID_v0.vi ‏97 KB

Hi andyabs,
I'm not getting the same error you are when running this VI. How are you setting up your channels and tasks in the DAQ Assistants? It seems like this may be the source of your error.
You should also take a look at Section 4 of the attached White Paper. Using DAQmx functions rather than the DAQ Assistant will give you a bit more functionality with your program. Let me know if you have any questions about this example.
Control Applications with Data Acquisition Hardware: http://www.ni.com/white-paper/9086/en/#toc4
Myriam
Applications Engineer
National Instruments

Minimize the count of cubes when there are more number of cubes

Hi All:
Need your inputs on reducing the count of cubes when there are more number of cubes in any project.
On which basis we can downsize the count of cubes? For example,if there 30 cubes in same region,I want to reduce them to ,say 20 cubes.
Please share your thoughts.
Regards,
Upendra.

You need to evaluate all of your existing cubes and see if there is any overlap in dimensionality that would allow you to combine cubes. Such as if one is actuals and another cube is budget for the same data, possibly you can combine those. Or as someone else suggested based on Business Units (If the other dimensions allow it. Perhaps conflicting Account or other structures won't allow it. )
However you have to then balance combining cubes against larger cube size which could impact calc time and other performance issues.
Its all a balancing act.
Look at where it makes sense to combine them. Only you know your environment enough to state if they can be combined or not.

Number of threads for imap process

What is the resonable nimber of threads for imap procerss?
I have 2 impad procrss and number of threads more then 200!!!
I think this is to much for 700-900 concurrent impa connection&
What is the reasons for this big number

Actually, your configuration sounds pretty close to correct for your load.
We allocate up to 250 threads, as many clients connect more than one time per session, and open more than one folder at a time.
I would not mess with the number of threads.
If most of your users have small numbers of messages and folders, you may be able to get away with a single imap process, but if you have several processors, two is fine, too.
Is there a problem or error situation you're trying to work around? What exact version of Messaging do you use? (imsimta version gets that)

Number of Threads Question

Hi,
I'm working on a pretty big program and i need a lot of threads to make it run well (it's an audio player and, of course, i don't need people to wait when, for example, they save a playlist). So, i put some of these tasks on different threads, plus others that i didn't need for making the program run faster, but they are there just to make the program run. Now i have at least 4 threads running at the same time and, in some cases, there are even 6 or 7. Is it too much? It's my first big program (it's just for fun but i still want it to run well) and i didn't need so many threads before. What's the maximum number of threads you'd recommend me to have?
Thanks.

Anakin1989 wrote:
Hi,
I'm working on a pretty big program and i need a lot of threads to make it run well (it's an audio player and, of course, i don't need people to wait when, for example, they save a playlist). So, i put some of these tasks on different threads, plus others that i didn't need for making the program run faster, but they are there just to make the program run. Now i have at least 4 threads running at the same time and, in some cases, there are even 6 or 7. Is it too much? It's my first big program (it's just for fun but i still want it to run well) and i didn't need so many threads before. What's the maximum number of threads you'd recommend me to have?
Thanks.Are you experiencing any problems with the number of Threads you have? If not, stop worrying. On the other hand, are you sure you need all of them? There are alternatives to Threading, such as SwingWorker and Timer that might be able to help you out.
One thing you better make sure of is that you're not doing any Swing work off of the EDT, or doing things like sleeping or time-consuming work on the EDT.

More than one thread in Openmp

Hello Friends,
I am trying to paralelize a great c program and now I am having problems for getting more than one thread.
I tried it inside my code, at first I put omp_set_dynamic(0) and after I put omp_set_num_threads(4), but I got only one thread.
These functions worked fine with a simple hello world code and I got to run 4 threads, but in my big code they didn't work. Could anybody help me? Any hint would be very good!...
Regards,
Glauber

Hi,
Thanks, It's a big code and was divided in 9 files, even so, I will post its most important parts.
That functions are called from sequential part before the parallel region as you will can see below:
// global.h
#if defined (OMP_ENABLED)
#     include "omp.h"
#endif
// main.c
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <math.h>
#include <string.h>
#include <stdarg.h>
#include "global.h"
#ifdef USE_READLINE
# include <readline/readline.h>
# include <readline/history.h>
static char **readline_completion(const char *, int, int);
#endif
# if defined (OMP_ENABLED)
int           proc_id; /* process ID (0, 1, ..., num_procs-1) */
int           num_procs; /* number of active processors */
# endif
int main (int argc, char *argv[])
     int i;
# if defined (OMP_ENABLED)
omp_set_dynamic(0);
omp_set_num_threads(4);
num_procs = omp_get_num_threads();
printf("Num procs: %i\n",num_procs);
if( num_procs <= 1)
exit(0);
# endif
//mcmc.c
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <math.h>
#include <string.h>
#include <ctype.h>
#include <limits.h>
#include <stdarg.h>
#include "global.h"
int RunChain (long int *seed)
     int          i, j, k, n, chn, stopChain;
     for (chn=0; chn<numLocalChains; chn++)
#if defined (OMP_ENABLED)
#pragma omp parallel for private(chn)
#endif
          for (chn=0; chn<numLocalChains; chn++)
If you need something else, please, return me
Thanks in advanced
Glauber

Do more Number of Threads reduce task duration

Similar Messages

Maybe you are looking for