Can i link a StreamTokenizer with a String Tokenizer?

Question

As recently informed, I now know that I can treat delimiters as tokens using StringTokenizer. My next question, is how do I use a StringTokenizer with a StreamTokenizer to let me read a word (eg cat) as a token and also a "," as a token? The reason being that I would like to be able to use the 'end of file' and 'end of line' flags in the StreamTokenizer but also would like to treat certain delimiters as tokens using the StringTokenizer!
If this is not possible, would I need to use the StringTokenizer and treat a space(" ") as a token (or delimiter)?
If I read a token with the StringTokenizer can I expect the StreamTokenizer to read the next token after that, or is that wishfull thinking?
The following code was used just to read words as tokens. I would like to incorporate the StringTokenizer to set delimiters as tokens to be read. How??????
try
    File file = new File("test.txt");
    FileInputStream stream = new FileInputStream (file);
    InputStreamReader reader = new InputStreamReader (stream);
    BufferedReader buffer = new BufferedReader (reader);
    StreamTokenizer tokenize = new StreamTokenizer (buffer);
    tokenize.nextToken();
    while (tokenize.ttype != StreamTokenizer.TT_EOF)
        data = tokenize.sval;
        tokenize.nextToken();
catch(IOException e)
    System.out.println("Error in file input:" + e.toString());
    }

Answer 1

I don't think you should do it that way, it can get a little nasty. Just use a BufferedReader to read one line in at a time, then use a StringTokenizer to break it up. When a reader has reached the eof, the readLine returns a null.