How to Exclude Java Reserved Words Using StringTokenizer

Question

Hi, i'm trying to exclude identifiers and java reserved words using StringTokenizer from an input file outputting the remaining words alphabetically ordered with the line, on witch they occur, in front of the word.
So far my code does all that, except exclude those words. I'm kind on a dead end... Any suggestions?
The code:
public class Interpreter{
    /* Uses a Binary Search Tree      */
    public static void main(String args[ ]) {
        String path = null;
        for (int j = 0; j < args.length; j++) {
            try {
                // Step 1, read the input file
                path   = args[j];
                FileReader input = new FileReader(args[j]);
                BufferedReader br = new BufferedReader(input);
                String line;
                int count = 1;
                String word;
                // Step 2, create an empty Tree
                BST tree = new BST();
                while((line = br.readLine())!=null){
                    StringTokenizer str = new StringTokenizer(line);
                    while (str.hasMoreTokens()){
                        word = str.nextToken();
                        tree.insert(word,count);
                    count++;
                // We're done, print out contents of Tree!
                tree.print();
                // Error Handling
                //check for IO problem with bad file
            } catch (IOException e) {
                System.out.println("There was a problem with the filename you entered.");
                //check for no filename entered at command prompt
            } catch (ArrayIndexOutOfBoundsException e) {
                System.out.println("Please include a filename when running this program.");
}Edited by: Redol on Dec 12, 2007 8:32 PM
Edited by: Redol on Dec 12, 2007 8:33 PM

Answer 1

use split instead of tokenizer.
public String[] splitString( String line, String delim )
    String[] tokens = null;
    if( line != null )
        tokens = line.split( delim );
    return tokens;
}