Questions on StringTokenizer, trim(), and parsing.

Question

im doing an assignment that requires parsing and havent had that much practice with parsing regular text from a file, ive been doing mostly parsing from a html. my question is if im parsing a line and i need 2 different types of information from it, should i just tokenize it twice or do it all at once, assuming the text format is always the same. for example
String input= "this is a test[10, 9, 8, 7, 6, 5, 4, 3, 2, 1]";if parsed correctly with StringTokenizer it would be 4 Strings(this is a test) and 10 ints for the countdown of numbers. now since the Strings doesnt have a delimiter such as a comma i can use the default delimiter which is the whitespace but then would that mean i would have to parse that same String twice since the numbers has "," has a delimiter? also should i worry about the whitespace that is separating the numbers after the comma? i did a small driver to test out the trim() using this and both outputs were the same. this may be a dumb question but if i call the trim() it eliminates the white space right, therefore i can just set "," as my delimiter, question is why is my output for both Strings the same?
    String input= "this is a test[10, 9, 8, 7, 6, 5, 4, 3, 2, 1]";
    String trimmed = test.trim();
    System.out.println(input);
    System.out.println("\n" + trimmed);SORRY if its confusing im trying not to reveal too much of the hw assignment and just get hints on parsing this efficiently. Thanks in advance

Answer 1

similar example on how to parse out the numbers with
"," as a delimiter. thanks in advanceThe following is a simple recursive descent parser to parse a comma delimited list of numbers "(1, 2, 3, 5, 6)". The grammar parser by this code is
START -> LPAREN LIST RPAREN
LIST -> NUMBER TAIL
TAIL -> COMMA LIST | LambdaThe nonterminals NUMBER, LPAREN, RPAREN, and COMMA are defined by the regular expressions in the code.
Lexical analysis is done by the function advanceToken(). It stores the next token in the variable "lookahead" for further processing. The parse tree is represented recursively through the functions start(), lparen(), rparen(), list(), number(), tail(), comma(), which match the corresponding symbols in the grammar. Finally, translation is done in the function number(). All it does is put the numbers it finds into the List intList. you can modify it to your needs.
This code originally parsed simple arithmetic expressions, but it took only ten minutes to parse lists of integers. It's not perfect and there are several obvious improvement that will speed up performance, however the strength of the design is that it can be easily changed to suit a variety of simple parsing needs.
import java.util.regex.*;
import java.util.*;
public class RDPParenList {
    private static final Pattern numberPat=
                    Pattern.compile("([1-9]\\d*)|0");
    public static final Object NUMBER = new Object();
    public static final Pattern commaPat = Pattern.compile(",");
    public static final Object COMMA = new Object();
    public static final Pattern lparenPat=
                    Pattern.compile("\\(");
    public static final Object LPAREN = new Object();
    public static final Pattern rparenPat=
                    Pattern.compile("\\)");
    public static final Object RPAREN = new Object();
    public static final Token NULLTOKEN = new Token(null, null);
    String input;
    String workingString=null;
    Token lookahead=NULLTOKEN;
    List intList = new ArrayList();
    /** Creates a new instance of RecursiveDescentParse */
    public RDPParenList(String input) {
        this.input=input;
    public void parse(){
        workingString=input;
        advanceToken();
        start();
        if(! "".equals(workingString))
            error("Characters still remaining in input '" + workingString + "'");
    private void advanceToken() {
        // calling advanceToken must give a token
        if("".equals(workingString))
            error("End of input reached unexpectedly");
        // prune the old token, and whitespace...
        if(lookahead != NULLTOKEN){
            int cutPoint = lookahead.symbol.length();
            while(cutPoint < workingString.length() &&
                    Character.isWhitespace(workingString.charAt(cutPoint))){
                ++ cutPoint;
            workingString=workingString.substring(cutPoint);
        // Now check for the next token, starting with the null token...
        if("".equals(workingString)){
            lookahead=NULLTOKEN;
            return;
        Matcher m=numberPat.matcher(workingString);
        if(m.lookingAt()){
            lookahead=new Token(m.group(), NUMBER);
            return;
        m=commaPat.matcher(workingString);
        if(m.lookingAt()){
            lookahead=new Token(m.group(), COMMA);
            return;
        m=lparenPat.matcher(workingString);
        if(m.lookingAt()){
            lookahead=new Token(m.group(), LPAREN);
            return;
        m=rparenPat.matcher(workingString);
        if(m.lookingAt()){
            lookahead=new Token(m.group(), RPAREN);
            return;
        error("Error during lexical analysis. Working string: '" +
                       workingString + "'");
    private void start() {
        lParen(); list(); rParen();
    private void lParen(){
        if(lookahead.attrib == LPAREN){
            advanceToken();
            // OK. Do nothing...
        else error("Error at token '" + lookahead.symbol + "' expected '('");
    private void rParen(){
        if(lookahead.attrib == RPAREN){
            advanceToken();
            // OK. Do nothing...
        else error("Error at token '" + lookahead.symbol + "' expected ')'");
    private void list() {
        number(); tail();
    private void number() {
        if(lookahead.attrib == NUMBER){
            // Do something with the number!
            try{
                intList.add(new Integer(lookahead.symbol));
            catch(NumberFormatException e){
                // This shouldn't happen if the lexer is working...
                e.printStackTrace();
                error("Unknown Error");
            advanceToken();
        else error("Error at token '" + lookahead.symbol + "' expected a number");
    private void tail() {
        if(lookahead.attrib == COMMA){
            comma(); list();
        else {
            // Lambda production
    private void comma() {
        if(lookahead.attrib == COMMA){
            advanceToken();
            // OK. Do nothing...
        else error("Error at token '" + lookahead.symbol + "' expected ','");
    private void error(String message){
        System.out.println(message);
        System.exit(-1);
    public static class Token{
        public Token(String symbol, Object attrib) {
            this.symbol=symbol;
            this.attrib=attrib;
        public String symbol;
        public Object attrib;
    public static void main(String []args){
        if(args.length == 0)
            return;
        System.out.println("\nParse String: " + args[0]);
        RDPParenList p=new RDPParenList(args[0]);
        p.parse();
        System.out.println("OK!");