Changing a Single Threaded app into Multi Threaded app

Question

Hi Java Experts
I made a web crawler that goes out to a particular web site and starts from there to crawl throughout the internet -- it's working fine . it only has 1 thread -- which is the main class which implements runnable.
I'm interested in making this solution multithreaded in hopes that I can get greater performance. -- since I believe the limiting factor is probably the speed at which URLs are returned etc --
MY QUESTION: how would you guys turn such a crawler into a multithreaded application.
my main class -- that does all the real work (with the help of some other classes works like this )
getTheNextURL(String urlString)
//step 1: check if it's a valid url
//step 2: check if it's a url we are interested in
//step 3: go to the url and gather all the links on that page put them in an array list
//step 4: loop through the array list
//call getTheNextURL(urlname) inside of the step loop
Does anyone see any way to make this multithreaded ? for increased urls looked at I was hoping i could possible multiply the url search speed by a factor of the threads.
Thanks in Advance for any suggestions that can be offered.
stev
i'm also aware of the problem with my recursive calls as read in the oriely java perfomance tuning book i'm looking to resolve thatas well with the most optimum algorithm

Answer 1

thanks for the suggestion sounds simple enough.
Since you made a great suggestion , what do you think about the recursion problem in my design.
Here's the problem
let's say the first web page it finds is a html page with the following links
link a
link b
link c
link d
It places these links in an ArrayList.
It loops through the ArrayList with an iterator and follows all the urls. starting with link a. On link a , the following link are found as seen below and placed in an ArrayList again.
link 1
link 2
link 3
link 4
then it will call the method again , which puts all the links found in an ArrayList and loops through it to follow the first.
This repeats over and over until it goes to the maximum depth specified , then it returns to the next iteration and looks through that. That's how it works in detail.
Every recursive call is an addition to the operating stack of the vm, so it's concievable that i may have 1000s of recursive calls stored on the stack each holding 1000s of ArrayList objects that it will consider.
I'm trying to figureout a better solution for this recursion to reduce the calls on the stack and implement multithreading at the same time.
Any ideas?
ps .
---- i would also like to be able to manually cancel threads that get caught in some infinite loop trick web pages out there.