How to decide the number of threads?

Dear all,
Suppose we have a multi-thread application, and we know that each thread is doing the same kind of work. What is the reasonable number of threads that we should set at begining.
It seems that a thread is expensive, so we should use only 1 thread. (wrong)
It seems that we should use 100000000000 threads. (wrong)
I would like to know your opnion about the above questions.
Regards.
Pengyou

pengyou wrote:
Thanks for the two answers.
In fact, I have made a crawler to crawl internet sites. I would like to know in practice should I:
1. Run the application to crawl the Internet site one by one;
2. Use multi-threads to crawl n sites in prarallel; (If one thread failed down, what to do?);
3. Run the application n times to crawl n sites.You will likely be spending a bunch of time in blocking I/O, so should gain a bunch by adding multiple threads. However, please be nice to the web hosts:
- Put all requests to the same domain into a single thread. This prevents your crawler from stealing too much bandwidth from the web host, making it harder for other real live people from using the site. Some sites will actually detect such hits and treat them as attacks and block them. One way to make sure you don't make too many simultaneous requests to the same domain would be to run all of a domain's requests into a single thread.
- To get the right amount of Threads, you will need to profile. Run the crawler with different amount of threads and see what settings make the best use of CPU time (fill CPU time to nearly 100%) without excessive thread switching.
How to handle failed threads? I would suggest using a Thread Pool (ThreadPoolExecutorService) and let the thread pool handle thread generation / death. You would have to make sure your code handles exceptions intelligently, and synchronizes properly still.

Similar Messages

  • How to decide the number of project for CHARM?

    We want to use only one project for all normal change requests(CR).
    But some CRs are very quickly moved into the PRD while some others are slower.
    Should we create 2 projects : one for fast moving corrections and another for slower ones?
    Above is for normal corrections only. The shortcoming of above plan is that it hard to know the CR is going to be fast or not.
    U r gent correction:  We have a dedicated project for u r gent corrections which are usually moving very fast.
    Could you tell us your practice on this?
    Thanks and regards.

    Dear Sneha:
    So roughly speaking, 3 projects are needed for CHARM:
    one for fast moving normal corrections
    one for slow moving normal corrections
    one for u r gent corrections
    Would you please confirm?
    Thanks!

  • How to find out the number of threads created under java vm at runtime

    our application seems to have hit the max number of threads that can be created under vm and the vm will just hang after that. that behavior seems to be consistent from an article that I read earlier.
    I wonder any way that java app can find out the number of threads currently created under vm?
    thanks in advance!

    If you are not starting an extra thread group you can use the Thread.activeCount method. But this willnot return all the threads, as there are some jvm threads, i.e. garbage collector. Also if you are running on a Unix type operating system then you can send a signal to the jvm to give you a thread dump, i think the signal is sighup.
    If the application is hanging i would of thought that you have a deadlock situation rather then the JVM not being able to create new threads, roughly how many threads should be running ??

  • How to increase the number of HTTP Worker Threads?

    Hi Buddy,
    I am using the lateset NW AS Java. It's a DEV system and the number of HTTP Worker Threads is 5.
    Do you know how to increase the number?
    What is the number in a Production System?
    Thank you!

    Parameter name: FCAServerThreadCount
    It's under Service->http.
    Use config tool for NW70.
    For NW71 and above use NWA and config tool.
    Also refer this -
    http://wiki.sdn.sap.com/wiki/display/JSTSG/%28JSTSG%29%28Web%29FAQ-FCAServerThreadCount

  • How do we decide the number of data base servers and application servers ?

    Dear Gurus,
    How do we decide the number of data base servers and application servers in BW ?
    How does this number help in data load balance and reporting needs ?
    Any hard n fast rule in the number of servers ?
    Regards
    Mohan

    Hi Mohan,
    Check the number of application server in the T-code SM66.... and also u can check the it in Sm50 on appplication server basiss./.
    From Sm37 u can see the application server Push button.. if u click on that also.. u can see the number of application servers in the system..
    Get into the individual application server and check how many Dialogue and background WP's are free,,
    If u see that the considerable number of WP are free(Waiting for task).. then u can shedule the loads..
    If most of the WP are  busy then u need to wait to trigger ur loads to maintain the load balancing..
    If one Application server is too busy then u can check the other Application Servers in the systems..
    And u can shedule the loads on a specific application sever level as well using the PC..
    Thanks
    I hope this is clear to u..
    Assign points if this helps

  • How to find the number of references to an object in a big application

    How to find the number of references created for a given object in a big application environment.
    That means, if i give any object name (of my application) as input, then how can i find the[b] number of references created for that particular object ??

    Please don't post the same question multiple times.
    I've answered your question in the other thread here:
    http://forum.java.sun.com/thread.jspa?messageID=4312939

  • How to set the number of seconds a servlet is allowed to run

    I use JSP to generate a report, but it will take about 10 minutes to search.
    IE Client screen displays an error message what is "Cannot find out your page" after 8 minutes. How to set the number of seconds a servlet is allowed to run.

    It's not a matter of how long the servlet is running... it's the browser timing out because the servlet hasn't responded to its request.
    You have several options:
    1) "Browser Pinging"
    Your servlet sends some small data which can be either seen or unseen (html comments, hidden chars, etc) by the user at short intervals while your report is running. When the report is finished, the browser will not have timed out because it has been "snacking" on those small bits of data which tell the browser its original request was both heard and being handled. I don't think there is any timeout in IE as long as it receives data continually (or at least before its own timeout mark over and over again...)
    2) Multithreaded processing
    This would probably be a better approach. Have the report run in a separate thread running on the server. You'd want to store a reference to this executing report in the user's session. Instead of making the browser wait for the report to be finished, have the servlet check the user's session to see if a report exists and is running. If one does not exist, create one and start its execution. If one does exist, and is still running, print a "please wait" type of message OR an animation, etc... along with some javascript which will reload the page every few seconds. If the page reloads and the servlet sees that the report is finished, it can then display it to the user.
    Hope this helps,
    -Scott

  • How to calculate the number of sent/received emails of a certain domain

    Thank you for what you have helped me with!
    How to calculate the number of sent/received emails of a certain domain in a certain period? It is Messaging Server 5.2, Directory Server 4.2. Is there a log option for this?
    Thank you.

    Not sure where you find, "LOG_MESSSAGE_ADD". I don't actually find this option in the documentation.
    The domains that mails are coming from and being sent to are certainly logged in the normal mail.log, so why mess with additional logging options? If you're talking about "LOG_CONNECTION", I actually see no additional data that is useful to you.
    If you decide to change the option.dat, you do indeed need to
    imsimta cnbuild (note, it's not cnrebuild)
    imsimta restart
    If I were facing the same issue, I'd be looking at the log parsing perl script, and simply modifying it to do what I wanted.

  • Is there any way to limit the number of Threads running in Application(JVM)

    Hello all,
    is there any way to limit the number of Threads running in Application(JVM)?
    how to ensure that only 100 Threads are running in an Application?
    Thanks
    Mohamed Javeed

    You should definitely use a thread pool for this. You can specify maximum number of threads that can be run. Be note that the thread pool will only limit the number of threads that are submitted to it. So donot call "Thread"s start() method to start thread on your own. Submit it to the pool. To know more, study about executor service and thread pool creation. Actually it will not be more than 20 line code for a class and you might need maximum of 2 such classes, one for threadPool and other one for rejection handler (if you want).
    Just choose the queue used carefully, you just have to pass it as a parameter to the pool.
    You should use "Bounded" queue for limiting threads, but also be careful in using queues like SynchronizedQueue as the queue will execute immediately the threads submitted to it if maximum number of threads have not been running. Otherwise it will reject it and you should use rejection handler. So if your pool has a synchronized queue of size 100, if you submit 101th thread, it will be rejected and is not executed.
    If you want some kind of waiting mechanism, use something like LinkedBlockingQueue. What this will do is even if you want 100 threads, you can specify the queue's size to be 1000, so that you can submit 1000 threads, only 100 will run at a time and the remaining will wait in the queue. They will be executed when each thread already executing will complete. Rejection occurs only when the queue oveflows.

  • Limit the number of threads

    Hello,
    I am developing an application which has to execute untrusted Java source code. The Java security manager is helpful in preventing unauthorized network and local file system access, but as far as I can tell, it can not limit the number of threads created by the untrusted application.
    How would I limit the number of threads an untrusted Java application is allowed to create?
    Jure
    Edited by: zbontarj on Feb 16, 2009 9:46 AM

    zbontarj wrote:
    Create a custom ThreadGroup will allows only a limited number of threadsI can't see any methods in ThreadGroup that limit the number of threads. There is a method that returns the number of active threads in the group, but that doesn't help me out much.ThreadGroup is not designed for easy extensibility.
    You can either take a copy and modify it to take a thread limit. (You can override the default implementation by using -Xbootclasspath/p:path or adding a jar to lib/endorsed however this may contravene the JDK license.)
    Or you can write a thread which monitors threads in the thread group every few seconds.
    Or you can use the SecurityManager to check the threads in the thread group. The SecurityManager is called each time a thread is added and it can check the threads in the current thread group.

  • How to delete the number saved in CALL FORWARDING list?

    how to delete the number saved in CALL FORWARDING list? seems like the number cannot be deleted in CALL FORWARDING list, i tried several times...
    he thing is, here are three numbers in the call forwarding list, one of them were set up by mistake, and none of them can be deleted.......it brothers me all the time. i already turn it off, but the number list is still there...

    Hello dennis130915 and welcome to the BlackBerry Support Community Forums.
    The image you have uploaded is unable to be displayed.
    Call Forwarding is a carrier controlled feature. Most likely, the number you see could be for the Voice Mail server.
    It's advised you call your mobile service provider to further assist with this feature.
    Thanks!
    -HMthePirate
    Come follow your BlackBerry Technical Team on twitter! @BlackBerryHelp
    Be sure to click Kudos! for those who have helped you.Click Solution? for posts that have solved your issue(s)!

  • How to find the number of data items in a file written with ArryToFile function?

    I have written an array of number in 2 column groups to a file using the LabWindows/CVI function ArrayToFile...Now if I want to read the file with FileToArray Function then how do I know the number of items in the file. during the write time I know how many array items to write. but suppose I want the file to read at some later time then How to find the number of items in the file,So that I can read the exact number and present it. Thanks to all
    If you are young work to Learn, not to earn.
    Solved!
    Go to Solution.

    What about:
    OpenFile ( your file );
    cnt = 0;
    while ((br = ReadLine ( ... )) != -2) {
    if (br == -1) {
    // I/O error: handle it!
    break;
    cnt++;
    CloseFile ( ... );
    There are some ways to improve performance of this code, but if you are not reading thousands of lines it's quite fast.
    After this part you can dimension the array to pass to FileToArray... unless you want to read it yourself since you already have it open!
    Proud to use LW/CVI from 3.1 on.
    My contributions to the Developer Zone Community
    If I have helped you, why not giving me a kudos?

  • How to find the number of entries in a master data table

    Hi Experts,
    I am trying to find the entries in 0CUSTOMER master data.
    BW>LISTCUBE>Data target: 0CUSTOMER and selected the fields that I need.
    I would like to know how to find the "number of entrees". I tried to run the SUM for a count field, but it is taking forever as there are huge number of records .

    Hi Dev,
    Go to the change/display mode of the info object (0CUSTOMER) in your case. Go to the Master data/Text tab. Here you will find the master data tables according to your settings (P orQ or X or Y). Double click on the table name and it will take you to the SE11 display. From there, you can check the number of records as you do in any transparent table.
    Hope this helps.
    Thanks and Regards
    Subray Hegde

  • How to find the number of links in a website

    hi every body
    can any one clear my doubt
    my doubt is that
    in a website how to find the number of links the hole website contains
    is there any navigation tool to find that
    plz help me yar
    i am waiting for the reply

    If you're talking about commercial sites (sites you don't have the source code for) I'm not sure.
    If you want to count the links in site that you have the source code for, it should be relatively easy to write a Java application to do it. All you really need to do is have a list of the files you want to count links in, read each file line by line and check for the existance of the <a> tag in each line. Anywhere their is an <a> tag there is a link.
    Hope this helps.

  • How to find the number of times method being called.....

    hi,
    can any one pls tell me how to find the number of times the method being called......herez the example....
    Refrence ref = new Refrence();
    for(int i = 0;i < arr.length; i++){
    if(somecondition){
    ref.getMethod();
    here i want to know how many times the getMethod() is calling...Is there any method to do this.. i have seen StrackTraceElement class..but not sure about that....pls tell me the solution....

    can any one pls tell me how to find the number of times the method being called......
    herez the example.... http://www.catb.org/~esr/faqs/smart-questions.html#writewell
    How To Ask Questions The Smart Way
    Eric Steven Raymond
    Rick Moen
    Write in clear, grammatical, correctly-spelled language
    We've found by experience that people who are careless and sloppy writers are usually also careless and sloppy at thinking and coding (often enough to bet on, anyway). Answering questions for careless and sloppy thinkers is not rewarding; we'd rather spend our time elsewhere.
    So expressing your question clearly and well is important. If you can't be bothered to do that, we can't be bothered to pay attention. Spend the extra effort to polish your language. It doesn't have to be stiff or formal ? in fact, hacker culture values informal, slangy and humorous language used with precision. But it has to be precise; there has to be some indication that you're thinking and paying attention.
    Spell, punctuate, and capitalize correctly. Don't confuse "its" with "it's", "loose" with "lose", or "discrete" with "discreet". Don't TYPE IN ALL CAPS; this is read as shouting and considered rude. (All-smalls is only slightly less annoying, as it's difficult to read. Alan Cox can get away with it, but you can't.)
    More generally, if you write like a semi-literate b o o b you will very likely be ignored. So don't use instant-messaging shortcuts. Spelling "you" as "u" makes you look like a semi-literate b o o b to save two entire keystrokes.

Maybe you are looking for

  • One tray for multiple button group items

    Hi, I have 3 button group items in my templaate ,in the propoerties of these button grps the tray is on .NOw I can see 3 trays when I execute my template instead I want to have one tray for these 3 button group items . How is this possible? THanks Sa

  • Bit of a B.T Dilema ?

    Hi All..I,m  kind of in a dilema  & puzzled It says on Open Reach website that my Exchanhe ( sg6 4pr ) is now Accepting Orders for FTTC  but  on the B.T Checker it says i,m not   due until March 31st , well i posted on here and was told it might be e

  • How do I open a word document in pages

    I brought a document created at work on microsoft word home and am trying to work with it on my mac using pages.  But when I open it a lot of the form is in disarray and I can't work with it.  Is there a way to open a word doument in pages without lo

  • How to get parked documents in KOB1

    Hi, Could you please tell me why the parked documents are does not showing in acutal line item reports? Is there a report that will show me actual/plan/commited and parked documents? Thanks in advance

  • Bug?  Rated photos won't laod on 5G iPod

    Does anyone else have this issue. I posted earlier about problems getting all my photos on my ipod and only unrated photos got loaded. So I got home and put no rating on all those same photos and they loaded. Can anyone else confirm this?