Help! where memory leak exist??

Question

I am new to java, and I wrote this web spider.But the programme is out of memory after downloading about 350 pages.It should download much more pages in theory.Is there memory leak?where?
I have been working on this for several days,but still not find out.Please help me!
package search;
import java.net.*;
import java.io.*;
import java.util.regex.*;
import file.FileOP;
public class SearchOneWeb extends Thread {
private String webName;//host name
private MutexList urlQueue;//MutexList is thread safe class I wrote
public SearchOneWeb(String webName, MutexList urlQueue) {
this.webName = removewww(webName);
this.urlQueue = urlQueue;
urlQueue.add("http://" + webName + "/");
start();
public SearchOneWeb(SearchOneWeb sow) {
this.webName = sow.webName;
this.urlQueue = sow.urlQueue;
start();
public void run(){     
try {
search();
} catch (Exception e) {
e.printStackTrace();
* remove www
* @param url
private static String removewww(String url) {
if(url != null) {
int index = url.indexOf("www.");
if (index != -1)
return url.substring(0, index) + url.substring(index + 4);
else
return url;
else
return null;
* make URL object
* @param url
private static URL makeURL(String url) {
if(url == null)
return null;
//only deal with HTTP
if (!url.toLowerCase().startsWith("http://"))
return null;
URL verifiedUrl = null;
try {
verifiedUrl = new URL(url);
if(verifiedUrl.getPath() == null)
verifiedUrl = new URL(verifiedUrl.toString() + "/");
} catch (Exception e) {
return null;
return verifiedUrl;
* test if this url belong to this host
* @param url
private boolean isInHost(URL url) {
boolean result = false;
if(url != null) {
String host = url.getHost().toLowerCase();
if(host.indexOf(webName) != -1)
result = true;
return result;
* down load web page
* @param url
* @return
private String downloadPage(URL url) {
String sURL = url.toString();
String pageContent = null;
try {
     HttpURLConnection con = (HttpURLConnection)url.openConnection();
     con.connect();
     BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream()));
String line;
StringBuffer pageBuffer = new StringBuffer();
while ((line = reader.readLine()) != null) {
pageBuffer.append(line).append('\n');
pageContent = pageBuffer.toString();
reader.close();
     con.disconnect();
} catch(IOException e) {
     System.out.println("error: can't connect to " + sURL);
return pageContent;
* write file
* @param url
* @param content
private void writeFile(String url, String content, String charSet) {
try {
FileOP.write(url, content, charSet);
} catch (IOException e) {
e.printStackTrace();
System.out.println("can't write file " + url);
* retrive links from page content
* @param pageUrl
* @param pageContents
private void retrieveLinks(URL pageUrl, String pageContents) {
//use regular expression to find links
Pattern pattern = Pattern.compile("<a\\s+href\\s*=\\s*[\"|']?(.*?)['|\"|>]", Pattern.CASE_INSENSITIVE);
Matcher m = pattern.matcher(pageContents);
int index1 = 0;
int index2 = 0;
String link = null;
String path = null;
String dir = null;
URL url = null;
while (m.find()) {
link = m.group(1).trim();
if (link.length() == 0)
continue;
if (link.charAt(0) == '#')
continue;
if (link.indexOf("mailto:") != -1)
continue;
if (link.toLowerCase().indexOf("javascript") != -1)
continue;
if (link.indexOf("://") == -1) {
if (link.charAt(0) == '/') {
link = "http://" + webName + link;
} else {
path = pageUrl.getPath();
dir = path.substring(0, path.lastIndexOf('/') + 1);
link = "http://" + webName + '/' + dir + link;
} else {
if (!link.startsWith("http"))
continue;
else {
index1 = link.indexOf('/', 7);
if (index1 == -1)
link = link + '/';
index2 = link.indexOf('#');
if (index2 != -1) {
link = link.substring(0, index2);
link = removewww(link);
url = makeURL(link);
if (url == null)
continue;
if (!isInHost(url))
continue;
if (urlQueue.contains(link))
continue;
urlQueue.add(link);
public void search() throws Exception {
if (urlQueue == null) {
System.out.println("urlQueue is null!");
return ;
String sURL = null;
URL url = null;
String pageContents = null;
while (urlQueue.size() > 0) {
     // Get URL at bottom of the list.
     sURL = urlQueue.getHead();
     url = makeURL(sURL);
     if (url == null)
continue;
     pageContents = downloadPage(url);
     if (pageContents != null && pageContents.length() > 0) {
writeFile(url.toString(), pageContents, "GBK");
retrieveLinks(url, pageContents);
* for test
* @param args
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
MutexList urlQueue = new MutexList();
SearchOneWeb sow = new SearchOneWeb("news.163.com", urlQueue);
sleep(10000);
SearchOneWeb sow1 = new SearchOneWeb(sow);
SearchOneWeb sow2 = new SearchOneWeb(sow);
}

Answer 1

You said:
I don't delete url from urlQueue,but the point is
that the urlQueue does't consume much memory.
You also said:
The size of urlQueue is tens of thousands when out of
memory,so it consumes several MB memory.
Are these statements both meant to be true at the same time?I meant that urlQueue was not the main reason for out of memory,enen though it consumed several MB memory.It is the strings saving page content that result to out of memory.These strings account for more than 30MB memory when out of memory.The point is why these strings are not GCd. I can not figure it out.
You don't need a queue element after you've processed
it, or even while you're processing it, so just keep
removing and processing the first item in the queue
until it is empty. That's how queues are normally
processed.I modified urlQueue.It keeps removing and processing the first item.But there is no improvement.The strings contaning page content are still not GCd.Why???