Java concurrency in vectors

Question

I am developing a simple web crawler program and am trying to store the results in two parallel vectors.
One function will update both vectors whilst the calling function will work its way through both. Eventually, they should both stop.
The problem I have though is that when 's = (String) itLinks.next ();' is called in the valPagesfunction, I receive the following:
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:449)
at java.util.AbstractList$Itr.next(AbstractList.java:420)
at LS3Q4WebCrawler.valPages(LS3Q4WebCrawler.java:154)
at LS3Q4WebCrawler.main(LS3Q4WebCrawler.java:445)
Is anybody able to tell me where I am wrong and how I can fix the program? I would rather not use threads if possible.
Many thanks
Martin
The code is below:
// Martin O'Shea, MSc AIS, Web Crawler Page for Labsheet 3 of module World Wide Web
// Technologies, Birkbeck College, University of London ID No 4939118511.
// Question 4.
// NOTE: All URL and file names in program LS3Q4CLWebCrawler.java are hardcoded and definitions of new
// Java Exception types has been avoided intentionally.
import java.io.*;
import java.util.*;
import java.net.*;
import java.text.*;
import javax.swing.text.*;
import javax.swing.text.html.*;
public class LS3Q4WebCrawler
    // Class instance variables.
    private Vector Instructions; // Records URLWithRobotsTxt's instructions.
    private Vector Pages; // Records pages as "parents".
    private Vector Links; // Records all "parent's" links as "children".
    private Vector PagesAlreadyVisited; // Records those pages already visited to avoid duplicate visits.
    private String URLRoot;
    private String URLWithRobotsTxt;
    private int PAGE_VAL_COUNT_ACHIEVED = 0;
    private final int PAGE_VAL_COUNT_NEEDED = 4; // Applies to all class instances as per following note.
    // NOTE: private final int pageValCount applies to all class instances and is set to four because.
    // i.     A page must be permitted by URLWithRobotsTxt.
    // ii.     A page's URL cannot be a relative address.
    // iii.     A page's URL must conform with URLRoot.
    // iv.     A page cannot be visited twice.
    public LS3Q4WebCrawler ()
    // Constructor.
        this.Instructions = new Vector ();
        this.Pages = new Vector ();
        this.Links = new Vector ();
        this.PagesAlreadyVisited = new Vector ();
          this.URLRoot = new String ("http://www.dcs.bbk.ac.uk/~kevin/webtech/");
        this.URLWithRobotsTxt = new String ("http://www.dcs.bbk.ac.uk/~kevin/webtech/robots.txt");
        this.PAGE_VAL_COUNT_ACHIEVED = 0;
    public LS3Q4WebCrawler (String s)
    // Overloaded constructor.
        this.Instructions = new Vector ();
        this.Pages = new Vector ();
        this.Links = new Vector ();
        this.PagesAlreadyVisited = new Vector ();
        this.URLRoot = new String (s);
        this.URLWithRobotsTxt = new String (s + "robots.txt");
        this.PAGE_VAL_COUNT_ACHIEVED = 0;
    public void validateCLParameters (String [] args)
    // Validates that there are no command line parameters.
        try
            if (args.length != 0)
                throw new IllegalArgumentException ();
        catch (IllegalArgumentException e)
            System.err.println ("ERROR: " + e);
            System.exit (1);
    public void getRobotInstructions ()
    // Reads URLWithRobotsTxt and extracts instructions from it.
    // The following code is adapted from "http://www.computing.dcu.ie/~cpahl/teach/ca597/Material/FileCon.html".
        try
            String s = new String ();
            URL url = new URI (URLWithRobotsTxt).toURL();
            BufferedReader br = null;
            InputStream is = url.openStream();
            br = new BufferedReader (new InputStreamReader (is));
            while ((s = br.readLine ()) != null)
                Instructions.addElement (new String (s));
        catch (MalformedURLException e)
            System.err.println ("ERROR: " + e);
            System.exit (1);
        catch (URISyntaxException e)
            System.err.println ("ERROR: " + e);
            System.exit (1);
        catch (IOException e)
            System.err.println ("ERROR: " + e);
            System.exit (1);
        // End of adaptation.
    public void displayRobotInstructions ()
    // Displays instructions from URLWithRobotsTxt on screen.
        System.out.println ("\n" + URLWithRobotsTxt + " contains the following instructions: " + "\n");
        String s = new String (" ");
        Iterator itInstructions = Instructions.iterator ();
        while (itInstructions.hasNext ())
            s = (String) itInstructions.next ();
            System.out.println (s);   
    public void valPages ()
    // Validates whether pages can be checked for links.
         String s = new String (" ");
         String t = new String (" ");
        // Root page.
            s = URLRoot;
            t = valPage (s);
        if (PAGE_VAL_COUNT_ACHIEVED == PAGE_VAL_COUNT_NEEDED) // Can root page be checked for links.
               getPageLinks (t);     
        PAGE_VAL_COUNT_ACHIEVED = 0;
        // Non-root pages.
        Iterator itLinks = Links.iterator ();
        while (itLinks.hasNext ())
             s = (String) itLinks.next ();
             t = valPage (s);
             if (PAGE_VAL_COUNT_ACHIEVED == PAGE_VAL_COUNT_NEEDED) // Can non-root page be checked for links.
                  getPageLinks (t);
             PAGE_VAL_COUNT_ACHIEVED = 0;
     public String valPage (String page)
     // Validates whether a page should be searched or not.
          // A page must be permitted by URLWithRobotsTxt.
          if (permittedByRobotsTxt (page))
               PAGE_VAL_COUNT_ACHIEVED ++;
          else
               PAGE_VAL_COUNT_ACHIEVED --;
          // A page's URL cannot be a relative address.
         if (URLRoot.equals (page)) // Assumes that root page is not a relative URL.
               PAGE_VAL_COUNT_ACHIEVED ++;     
          else
               page = valFormatRelativeAddress (page); // Remove surplus leading characters.
               if (hasPageRelativeAddress (page))
                    page = (URLRoot + page); // Amend page URL.     
               PAGE_VAL_COUNT_ACHIEVED ++;
         // A page's URL must conform with URLRoot.
         if (hasPageRootAddress (page))
               PAGE_VAL_COUNT_ACHIEVED ++;     
          else
               PAGE_VAL_COUNT_ACHIEVED --;
          // A page cannot be visited twice.
          if (! hasPageBeenVisited (page))
               PAGE_VAL_COUNT_ACHIEVED ++;          
          else
               PAGE_VAL_COUNT_ACHIEVED --;
          return (page); // Formatted URL of page to be visited.
     public Boolean permittedByRobotsTxt (String page)
     // Is page permitted by URLWithRobotsTxt.
          Boolean found = false;
          String s = new String (" ");
        Iterator itInstructions = Instructions.iterator ();
        while (itInstructions.hasNext ())
            s = (String) itInstructions.next ();
            if (s.indexOf (page) >= 0) // Current page.
                      found = false; // Not permitted.
            else
                 found = true; // Permitted.     
             if (found == false)
                  break;
        return (found);
     public Boolean hasPageRelativeAddress (String page)
     // Has page a relative address?
          if (page.indexOf (URLRoot) >= 0)
               return (false);
          else
               return (true);
     public String valFormatRelativeAddress (String page)
     // Removes surplus leading characters from relative address of a page.     
          int substringToChar = page.length ();
          int substringFromChar = 0;
          if (page.indexOf ("./") >= 0) // Allow first two characters.
               substringFromChar = 2;
          else if (page.indexOf (".") == 0) // Allow first character.
               substringFromChar = 1;
          else if (page.indexOf ("/") == 0) // Allow first character.
               substringFromChar = 1;
          page = page.substring (substringFromChar, substringToChar);
          return (page);
     public Boolean hasPageRootAddress (String page)
     // Validates if page's URL conforms to the root address.
          if (page.indexOf (URLRoot) >= 0)    
               return (true);
          else
               return (false);
     public Boolean hasPageBeenVisited (String page)
        // Validates whether a page has been visited already.     
             Boolean found = false;
             String s = new String (" ");
        Iterator itPAV = PagesAlreadyVisited.iterator ();
        while (itPAV.hasNext ())
             s = (String) itPAV.next ();
             if (page.equals (s)) // Current page.
                      found = true; // Visited.
            else
                 found = false; // Not visited.
             if (found == true)
                  break;
        return (found);
    public void getPageLinks (String page)
    // Validates a web page and extracts links from it.
        try
            URL url = new URI (page).toURL();
            URLConnection uc = url.openConnection ();
            Reader rd = new InputStreamReader (uc.getInputStream ());
            EditorKit ek = new HTMLEditorKit ();
            HTMLDocument doc = (HTMLDocument) ek.createDefaultDocument ();
            doc.putProperty("IgnoreCharsetDirective", Boolean.TRUE);
            ek.read (rd, doc, 0);
            HTMLDocument.Iterator it = doc.getIterator (HTML.Tag.A);
            while (it.isValid ())
                SimpleAttributeSet sas = (SimpleAttributeSet) it.getAttributes ();
                String link = (String) sas.getAttribute (HTML.Attribute.HREF);
                if (link != null)
                    // Add page to vectors.
                    Pages.addElement (new String (page));
                    Links.addElement (new String (link));
                it.next ();
            // Record that page has been visited.
            PagesAlreadyVisited.addElement (new String (page));
        catch (MalformedURLException e)
            System.err.println ("ERROR: " + e);
            System.exit (1);
        catch (URISyntaxException e)
            System.err.println ("ERROR: " + e);
            System.exit (1);
        catch (BadLocationException e)
            System.err.println ("ERROR: " + e);
            System.exit (1);
        catch (IOException e)
            System.err.println ("ERROR: " + e);
            System.exit (1);
        // End of adaptation.
    public void writeResults ()
    // Populates file "crawl.txt".
        try
            // No OuputStream object used for file "crawl.txt" because its location cannot
            // specifically be known.
            BufferedWriter bw = new BufferedWriter (new FileWriter ("crawl.txt"));
            bw.write ("The following pages and links were found: " + "\n\n");
            int i = 0;
            int j = 0;
            String currPage = new String (" ");
            String s = new String (" ");
            Iterator itPages = Pages.iterator ();
            while (itPages.hasNext ())
                currPage = (String) itPages.next ();
                if (! s.equals (currPage)) // Current page?
                    bw.write ("<Visited " + currPage + ">\n");
                    s = currPage;
                    i ++;
                bw.write ("   <Link " + Links.elementAt (j) + ">\n");
                j ++;
            bw.close ();
        catch (IOException e)
            System.err.println ("ERROR: " + e);
            System.exit (1);
    public static void main (String [] args)
    // Main program.
        LS3Q4WebCrawler w = new LS3Q4WebCrawler ();
        // Validate parameter entry from command line.
        w.validateCLParameters (args);
        // Get URLWithRobotsTxt instructions, display on screen.
        w.getRobotInstructions ();
        w.displayRobotInstructions ();
        // Validates pages and extracts links from them.
        w.valPages ();
        // Populate file "crawl.txt".
        w.writeResults ();
}[                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

Answer 1

No I'm saying the behavior of an iterator is undefined if you are modifying the collection that the iterator came from. Even if the modification wouldn't invalidate the iterator (as is probably the case in your problem), they are often designed to "fail-fast" which means throw that exception as soon as they detect a modification.
The queue on the other hand doesn't require an iterator, since you remove (dequeue) the first item each time. getPageLinks will work similarly, adding new pages to visit to the queue--it doesn't have to return a list. When the queue is empty, there are no more pages to visit (this is pretty much the general setup for breadth first search which is really what you're doing)
If you still want to keep the list of Links, then I'm recommending adding them to the permanent list after they've been dequeued.