StreamTokenizer code: Don't understand.

Question

Can anyone explain in dumbass language what is going on here? I mean, I know the end result, but I don't exactly understand what is going on. Little help?
char separator = ',';
StreamTokenizer tokenizer = new StreamTokenizer(new BufferedReader(new StringReader(str)));
tokenizer.resetSyntax();
tokenizer.wordChars('\u0000', (char)(separator -1));    // Everything is a word character.
tokenizer.wordChars((char)(separator + 1), '\u00ff');    // except for the separator.The first two lines are just there to give you a sense of context. :) My main problem is the three calls to the tokenizer methods. What on earth is going on there?
Thanks for any help. :)

Answer 1

char separator = ',';
StreamTokenizer tokenizer = new StreamTokenizer(new
BufferedReader(new StringReader(str)));
tokenizer.resetSyntax();All characters are "regular" for this tokenizer--none are considered "word" or "number" or "whitespace" characters.
tokenizer.wordChars('\u0000', (char)(separator -1));
// Everything is a word character. Everything from character 0 up to, but not including, the comma character is considered a "word" character.
tokenizer.wordChars((char)(separator + 1), '\u00ff');
// except for the separator.Everything after the comma, up to character 0x00ff is also a "word" character.The docs for StreamTokenizer say it can be used to break a stream up into words, numbers, comments, etc. We've defined that a comma is "ordindary" and everything else (up to char 255) is part of a word.