String tokenizer question

I need to parse strings of the form "token A ops token B" where ops can be <, >, =, != and output "tokenA ops tokenB"....(remove spaces in the tokens).
I can use all the ops as delimiters (though I cant specify '!='), and get the tokens and remove the spaces in them. Using StringTokenizer, I doubt if I can know the exact delimiter I encountered so that I can rebuild the string quickly. Sure, I can do an indexOf() for each of the ops to find the one used in original string. But I am curious to know if there is any better approach to do this (doesnt have to be string tokenizer approach)?

If the tokens always consist of "word characters" (letters, digits, and underscores) and the operators always consist of punctuation characters, you can do this: str = str.replaceAll("\\b\\s+\\b", "");

Similar Messages

Quick string tokenizer question

HELP!
I'm quite new to using string tokenizers in Java.
I'm trying to use a string tokenizer to read some data into a multi dimensional int array and then print it to the screen.
What I'm using so far is:
FileReader file = new FileReader("test.txt");
BufferedReader inputFile = new BufferedReader(file);
StringTokenizer data;
for(int i=0; i<LEARN; i++)
     data = new StringTokenizer(inputFile.readLine());
     for (int j=0; j<NEURONS; j++)
                             examples[i][j] = new int(data.nextToken().intValue);
for(int row=0; row<LEARN; row++)
     for (int collumn=0; collumn<NEURONS; collumn++)
                         screen.print(examples[row][collumn] + " ");
                         screen.flush();
}However, I'm getting errors!!
Could anyone help at all??
Cheers
Ross

However, I'm getting errors!!The messages that come with errors usually play a significant role in resolving them. Would you mind posting those messages?
Where do you define the variable "examples" ?
new int(data.nextToken().intValue);This is definitely wrong.

String Tokenizer in JSP

What is the best way to write a string tokenizer is a JSP page??
I am passing a vector to the JSP page which is being outputted on to the page and now I want to edit the contents of the vector. I need to tokenize the vector into a series of strings but I am making a hash of it so far.
Is there a better way to do this?

I'm not sure what your exact question is. StringTokenizer works with String classes not Vectors. Do you want to change the data in the Vector and leave the Vector in tact, or do you want to retrieve the data from the Vector and do something with it?
You can retrive the contents of the Vector into an Object array, or you can enumerate through the elements, or you can use the get method itself. It depends on what you need to do.

Problems with string tokenizer

okay heres my problem, i have a textarea displaying a program listing and i have to extract variables i.e public int something;
i have tried many different approaches but still the output is not what i was looking for. Can someone have a look at my code and check for mistakes that i cant find
public void createDataDictionary() {
          if(fullText.equals("")) return;//do nothing if no text exists
          String checkText = fileContents.getText();//store text area contents into string
          String dataType = "";
          String variable = "";
          String accessModifier = "";
          StringTokenizer str = new StringTokenizer(checkText," \n", true);
          fileCheckBox.setText("");//clear file check box
          while(str.hasMoreTokens()) {//loop while there are more tokens to tokenize
               try{
                    checkText = str.nextToken();
                    //check for comments
                    if((checkText.startsWith("//")) || (checkText.equals("//")) ||
                       (checkText.startsWith("/**")) || (checkText.equals("/**")) ||
                       (checkText.startsWith("*")) || (checkText.equals("*"))) {
                         isComment = true;
                    if((checkText.equals("\n"))) isComment = false;
                    if(!isComment)
                         //check for access modifiers
                         if((checkText.equals("public")) || (checkText.equals("private")) ||
                            (checkText.equals("protected"))) {
                                      accessModifier = checkText;
                                 }else {
                                      accessModifier = "";
                         //check for data types
                         if((checkText.equals("boolean")) || (checkText.equals("char")) ||
                                 (checkText.startsWith("String")) || (checkText.equals("int"))) {
                                   dataType = checkText;
                                   variable = str.nextToken();//get variable expression
                                   System.out.println(accessModifier + " " + dataType + " " + variable);
               }catch(NoSuchElementException nsee) {//not enough tokens to complete operation
                    JOptionPane.showMessageDialog(null, "End of file");
                    return;//break from method
     }here is sample text
private String seven = "help";
char five[];
// String here
//int found
public boolean okay
and here is the output
String
char
String
boolean
//note the space before each output
hope someone can help
thanx in advance

1. Why do you check to see if the token starts with
//, /*, etc, etc. Later you check if the token
equals private, public, protected. It couldn't
equal if it started with a comment.if the token starts with a comment i dont want to read the rest of the line, so i set the isComment flag to true
2. I strongly suggest that you do it line by line.
Perhaps a string tokenizer that only tokenizes by
lines, then within that loop, another that tokenizes
by whitespace. i take it you mean putting the text into a bufferedreader and then using readLine()?? Bit new to string tokenization as you can possibly see
i managed to get the small test text to work more or less as i wanted but when ever i load a large code listing, the results are erratic to say the least
heres a section of this code
private int textNum = 0;/**Integer used to hold the text position within any given line*/
     private int lineNum = 0;/**Integer used to hold the line number within any given file*/
     static boolean application = false;/**Used to track if applet is ran from a browser or JAR file*/
     static boolean fileOpened = false;/**Used to track if file has already been opened*/
     static boolean isComment = false;
     private char lCurve = '(';
     private char rCurve = ')';
     private char lCurly = '{';
     private char rCurly = '}';
     private char lSquare = '[';
     private char rSquare = ']';
     String fullText = "";and heres the output
public int textNum //should be private!!!!
int lineNum //missing private
boolean application //missing static
boolean fileOpened //missing static
boolean isComment //missing static
//all below missing private
char lCurve
char rCurve
char lCurly
char rCurly
char lSquare
char rSquare
String fullText //not there at all

How to retrieve IndividualStrings from a txt file using String Tokenizer.

hello can any one help me to retrieve the individual strings from a txt file using string tokenizer or some thing like that.
the data in my txt file looks like this way.
Data1;
abc; cder; efu; frg;
abc1; cder2; efu3; frg4;
Data2
sdfabc; sdfcder; hvhefu; fgfrg;
uhfhabc; gffjcder; yugefu; hhfufrg;
Data3
val1; val2; val3; val4; val5; val6;
val1; val2; val3; val4; val5; val6;
val1; val2; val3; val4; val5; val6;
val1; val2; val3; val4; val5; val6;
i need to read the data as an individual strings and i need to pass those values to diffarent labels,the dat in Data3 i have to read those values and add to an table datamodel as 6 columns and rows depends on the data.
i try to retrieve data using buffered reader and inputstream reader,but only the way i am retrieving data as an big string of entire line ,i tried with stringtokenizer but some how i was failed to retrive the data in a way i want,any help would be appreciated.
Regards,

Hmmm... looks like the file format isn't even very consistent... why the semicolon after Data1 but not after Data2 or Data3??
Your algorithm is reading character-by-character, and most of the time it's easier to let a StringTokenizer or StreamTokenizer do the work of lexical analysis and let you focus on the parsing.
I am also going to assume your format is very rigid. E.g. section Data1 will ALWAYS come before section Data2, which will come before section Data3, etc... and you might even make the assumption there can never be a Data4, 5, 6, etc... (this is why its nice to have some exact specification, like a grammar, so you know exactly what is and is not allowed.) I will also assume that the section names will always be the same, namely "DataX" where X is a decimal digit.
I tend to like to use StreamTokenizer for this sort of thing, but the additional power and flexibility it gives comes at the price of a steeper learning curve (and it's a little buggy too). So I will ignore this class and focus on StringTokenizer.
I would suggest something like this general framework:
//make a BufferedReader up here...
do
String line = myBufferedReader.readLine();
if (line!=null && line.trim().length()>0)
line = line.trim();
//do some processing on the line
while (line!=null);So what processing to do inside the if statement?
Well, you can recognize the DataX lines easily enough - just do something like a line.startsWith("Data") and check that the last char is a digit... you can even ignore the digit if you know the sections come in a certain order (simplifying assumptions can simplify the code).
Once you figure out which section you're in, you can parse the succeeding lines appropriately. You might instantiate a StringTokenizer, i.e. StringTokenizer strtok = new StringTokenizer(line, ";, "); and then read out the tokens into some Collection, based on the section #. E.g.
strtok = new StringTokenizer(line, ";, ");
if (sectionNo==0)
//read the tokens into the Labels1 collection
else if (sectionNo==1)
//read the tokens into the Labels2 collection
else //sectionNo must be 2
//create a new line in your table model and populate it with the token values...
}I don't think the delimiters are necessary if you are using end-of-line's as delimiters (which is implicit in the fact that you are reading the text out line-by-line). So the original file format you listed looks fine (except you might want to get rid of that rogue semicolon).
Good luck.

Need help for string tokenizer!

I need help in writing out the string tokenizer.. i've got all of the functions written except for the nexttokken function which i am stuck in. Can anyone help me out in writing the function?

An example of splitting string by comma
String str"A,B,C";
StringTokenizer st = new StringTokenizer(str,",");
while (st.hasMoreTokens()) {
System.out.println(st.nextToken());
This is just an example of usage; i would prefer to use split() function instead

Help!! i dont know how to make use of token from string tokenizer :(

Hi,
I would like to ask how to manipulate the token that are taken from string tokenizer. For example :
input = a b 1 f 986
supposed output:
a
b
1
f
986
How to make the output look like this?
Output = (a,b,f),(1,986)
Below is the source code of Algebra.java
import java.io.*;
import java.util.*;
* Read and print, using BufferedReader from System.in, onto System.out
public class Algebra {
public static void main(String[] av) {
try {
BufferedReader is = new BufferedReader(new InputStreamReader(System.in));
String inputLine;
System.out.print("Input =");
while ((inputLine = is.readLine()) != null) {
System.out.println("Output="+inputLine);
StringTokenizer st =
new StringTokenizer(inputLine, ", |");
while (st.hasMoreElements())
System.out.println("Token: " + st.nextElement());
break;
is.close();
} catch (IOException e) {
System.out.println("IOException: " + e);
Thank You

You can modify this to serve your goal.
BufferedReader is=null;
try {
is = new BufferedReader(new InputStreamReader(new FileInputStream("Data.txt")));
StreamTokenizer st = new StreamTokenizer(is);
int n=0,k=0,q=0;
String[] s=new String[20];
String[] s0=new String[10];
while(st.nextToken() != StreamTokenizer.TT_EOF) {
        switch(st.ttype) {
          case StreamTokenizer.TT_EOL:
            s[k] = new String("EOL");k++;
            break;
          case StreamTokenizer.TT_NUMBER:
            s0[q] = Double.toString(st.nval);q++;
            break;
          case StreamTokenizer.TT_WORD:
            s[k] = st.sval; // Already a String
             k++;
            break;
          default: // single character in ttype
            s[k] = String.valueOf((char)st.ttype); k++;
        } //end of while
   //   Some output processing here
}catch(IOException ioe){
   ioe.printStackTrace();
}catch(ArrayIndexOutOfBoundsException bx){
   System.err.println(bx);
}finally{
       try{      if(is !=null)     is.close();}catch(IOException ioe){;}
}

String Tokenizer/ Multiple Client app.

Hi all,
I am developing a multi user pure Java App.
My situation:
1. client request server for connection. > server accepted
2. client send his details (name*|*group*|*ID)
3. server receive the string, update itself and other client accordingly
My problem:
1. to send client detail (name*|*group*|*ID), String tokenizer is being used? and *|" serve as delimeter ?
2. how do create an statement that count the total user connected to server or leave the server
3. and how to assign an identifier to each connected client, thus server can perform differ operation based on the identifier.
Thanks Ya, if someone can provide me some suggestions
Happy Life

To answer your problems directly
1. Client Info: Yes, read all information from the client using an inputstream. convert that into a String, and use StringTokenizer (str, "|") to parse the information
2. # of clients: After a client has connected to your server, they will be automatically shunted to another port that that user alone communicates on. You should set up a new thread to listen to data being sent on that port. Keep these listeners in an array/vector/arraylist and use getSize()
3. The identifier could just be their port #

String tokenizer manipulation

Hi,
I would like to ask how to manipulate the token that are taken from string tokenizer. For example :
input = a b 1 f 986
supposed output:
a
b
1
f
986
How to make the output look like this?
Output = (a,b,f),(1,986)
How to make the program will be able to recognize and arrange the data into the output that i want.
Below is the source code of Algebra.java
import java.io.*;
import java.util.*;
* Read and print, using BufferedReader from System.in, onto System.out
public class Algebra {
public static void main(String[] av) {
try {
BufferedReader is = new BufferedReader(new InputStreamReader(System.in));
String inputLine;
System.out.print("Input =");
while ((inputLine = is.readLine()) != null) {
System.out.println("Output="+inputLine);
StringTokenizer st =
new StringTokenizer(inputLine, ", |");
while (st.hasMoreElements())
System.out.println("Token: " + st.nextElement());
break;
is.close();
} catch (IOException e) {
System.out.println("IOException: " + e);
Thank You

I'd make two arrays and fill them separately depending on whether you can parse the token as an int or not. Try running this code:
import java.io.*;
import java.util.*;
* Read and print, using BufferedReader from System.in, onto System.out
public class Main {
public static void main(String[] av) {
    int numbers[] = new int[20];
    String variables[] = new String[20];
    int numberPosition = 0;
    int variablePosition = 0;
    try {
        BufferedReader is = new BufferedReader(new InputStreamReader(System.in));
        String inputLine;
        System.out.print("Input =");
        while ((inputLine = is.readLine()) != null) {
            StringTokenizer st =
            new StringTokenizer(inputLine, ", |");
            while (st.hasMoreElements()) {
                String token = st.nextToken();
                try {
                    int i = Integer.parseInt(token);
                    numbers[numberPosition++] = i;
                } catch (NumberFormatException e) {
                    variables[variablePosition++] = token;
            System.out.print("(");
            for (int x = 0; x < variablePosition -1; x++)
                System.out.print(variables[x]+",");
            System.out.print(variables[variablePosition-1]+")");
            System.out.print("(");
            for (int x = 0; x < numberPosition -1; x++)
                System.out.print(numbers[x]+",");
            System.out.print(numbers[numberPosition-1]+")");
            break;
    } catch (IOException e) {
        System.out.println("IOException: " + e);
}Sample Output:
Input =a b 1 f 986
(a,b,f)(1,986)
press any key to exit...

Parsing and String tokenizer

I am trying to use String Tokenizer to extract fields from a file then do a calculation on some of the fields.
My problem is when I try to parse the strings that need the calculations I get error codes as shown at the end of post. I don't see why the parsing shouldn't work.( error codes are class expected and incompatable types)
// ParsingTextFile.java: Process text file using StringTokenizer
import java.io.*;
import java.util.StringTokenizer;
public class ParsingTextFile1 {
/** Main method */
public static void main(String[] args) {
   // Declare buffered file reader and writer streams
   BufferedReader brs = null;
   BufferedWriter bws = null;
   //declare tokenizer
   StringTokenizer tokenizer;
   // Declare a print stream
   PrintWriter out = null;
    // Five input file fields:total string, student name, midterm1,
    // midterm2, and final exam score
    String line=null;
    String sname = null;
   String mid1= null;
    String mid2 = null;
    String finalSc = null;
/ double midterm1 = 0;
    double midterm2 = 0;
    double finalScore = 0;
    // Computed total score
    double total = 0;
    try {
      // Create file input and output streams
      brs = new BufferedReader(new FileReader("in.dat"));
      bws = new BufferedWriter(new FileWriter("out.dat"));
     while((line= brs.readLine())!=null){
     tokenizer =new StringTokenizer(line);
      sname = tokenizer.nextToken();
      mid1 = tokenizer.nextToken();
      mid2 = tokenizer.nextToken();
      finalSc= tokenizer.nextToken();
      midterm1 = double.parseDouble(mid1)//this code not working
      midterm2 = double.parseDouble(mid2);)//this code not working
      finalScore = double.parseDouble(finalScore);//this code not working
      out = new PrintWriter(bws);
      total = midterm1*0.3 + midterm2*0.3 + finalScore*0.4;
      out.println(sname + " " + total);
    catch (FileNotFoundException ex) {
      System.out.println("File not found: in.dat");
    catch (IOException ex) {
      System.out.println(ex.getMessage());
    finally {
      try {
        if (brs != null) brs.close();
        if (bws != null) bws.close();
      catch (IOException ex) {
        System.out.println(ex);
}[\code]
errorsC:\j2sdk1.4.1_06\bin\ParsingTextFile1.java:43: class expected
    midterm1 = double.parseDouble(mid1);
                      ^
C:\j2sdk1.4.1_06\bin\ParsingTextFile1.java:44: class expected
    midterm2 = double.parseDouble(mid2);
                      ^
C:\j2sdk1.4.1_06\bin\ParsingTextFile1.java:45: class expected
    finalScore = double.parseDouble(finalScore);
                        ^
C:\j2sdk1.4.1_06\bin\ParsingTextFile1.java:43: incompatible types
found   : java.lang.Class
required: double
    midterm1 = double.parseDouble(mid1);
                     ^
C:\j2sdk1.4.1_06\bin\ParsingTextFile1.java:44: incompatible types
found   : java.lang.Class
required: double
    midterm2 = double.parseDouble(mid2);
                     ^
C:\j2sdk1.4.1_06\bin\ParsingTextFile1.java:45: incompatible types
found   : java.lang.Class
required: double
    finalScore = double.parseDouble(finalScore);
                       ^
6 errors
Process completed.

while(line.hasMoreTokens())
sname = tokenizer.nextToken();
   mid1 = tokenizer.nextToken();
mid2 = tokenizer.nextToken();
   finalSc= tokenizer.nextToken();
    midterm1 = double.parseDouble(mid1)//this code not working
     midterm2 = double.parseDouble(mid2);)//this code not working
   finalScore = double.parseDouble(finalScore);//this code not working
}if it doesn't work then you need to add delimiter Or the token that you are trying to convert into is not valid.

Which is lightweight array list or string tokenizer

which is lightweight array list or string tokenizer:
I am reading a flat file and doing some text parsing to find some records which matched my search criteria.
The flat file will have 5K records at the average and each records is in each line. (\n) is the delimiter to the string tokenizer.
My Doubt is:
Will it be good to manipulate all the records to the array list and then manipulating and achieve my task or
Will it be good if i do a just string tokenizer and achieve my task with out an arraylist.
Note: There is no unique fields in the records so no way to use hashmap/table.

DrClap wrote:
Faster? Actually the stated requirements were "lightweight" and "good". It guess I assumed what they usually mean to me.
However answering this a different way.
lightweight - your computer won't be heavier no matter which way you choose.
good - it will have no barring on your concious or your place in the after life.
good(2) - Write it the simplest way you think you can and worry about performance later if its a problem.

Looping through string tokenizer

Hi pple...
I am doing a project in jsp...i have some pblm in a form that contains string tokenizer method..let me make u all very clear..
Ex:in my databse i have stored country as us|uk|uae|india..
so wat my requirement is once i login,depending upon my id it should check the country..if the login id has one country den no pblm i can easily check a condition and i will divert the page..if the loginid has two countrys what should i do..
i have made the foll code snippet:
log_country=rs.getstring();database values(i.e...pipe separated country)
string tokenizer st=new stringtokenizer(log_country,"|");
while (st.hasMoreTokens())
     i=0;
     st=st1.nextToken();
     i++;
for(i=0;i<st.length;i++)
     if(st[i].equals("null"))
          st[i]="";
          else
          out.println(""+st[i]);
for(i=0;i<st.length;i++)
if(st[i].equals("us"))
{%>
<jsp:forward page="admin2.jsp" />
<%}
if the st array contains two countrys how can i get the values ...
st[0]>>for us
st[1]>>for uk..i hav assumed ,,but it never works
how can i get to know these two countrys and i want to forward to one page..thanks in advance

Friend use code tag to post your code.
Design like below will work
String log_country = "";
    try {
        log_country = rs.getString("Countrys");    // database values (i.e...pipe separated country)
    } catch (Exception e) {
    String[] countrys = log_country.split("[|]");
    for(int i=0; i<countrys.length; i++){
        if(countrys==null || countrys[i].equals("null") || countrys[i].trim().length()==0){
out.println("No Country"); // do whatever
} else{
out.println(countrys[i]); // do whatever

Using String Tokenizer on connection string

I have a database connnection string like so:
jdbc:oracle:thin:@<url>:<port number>:<database name>
I'm planning to use String Tokenizer to extract the database name. Is this possible. Where can I start?

I have a database connnection string like so:
jdbc:oracle:thin:@<url>:<port number>:<database
name>
I'm planning to use String Tokenizer to extract the
database name. Is this possible. Where can I start?You can tokenize on ":". Then put eack token on a list. Yourdatabase name will be last element of the list. Take a look at on StrnngTokenizer API.

Regarding string tokenizer

Hello all,
Can anybody help me to get the file name which is specified in the path like C:/programs/folder/example.txt" or "C:/programs/folder/test.txt" or "C:/programs/example/verify.txt" . The path name will change dynamically in my program.
What I want to do is to get the filename from the path using string tokenizer.
Thanks in advance.

I think I do. Why don't you try that code snippet out
and see that when you invoke File.getName it returns
you the file name with the path part stripped out.
Isn't that what you said you wanted?Even if that ability did not exist (but clearly a programmer should always use it if it is available) then it would be simply to substring the full path starting at last index of the current platform's file separator, and ending at the end of the path string.

Is there any string tokenizer that handles quotes?

I need a string tokenizer that handles quotes in LabVIEW.
Assume that there is a string, abc bcc="xxx yyy"
After tokenizing, token 1 is abc, and token 2 is bcc="xxx yyy".
The tokenizer should ignore white characters between quotes.
I use LabVIEW 2009 SP1.
Does anyone have a tokenizer VI?
Solved!
Go to Solution.

I expect that I can make something cleaner if I had a better understanding of your expected input, but this works with what I know.
The bottom loop looks for start & end quotation marks and pastes the elements together that the first loop shouldn't have (based on your requirement) taken apart.
Jim
You're entirely bonkers. But I'll tell you a secret. All the best people are. ~ Alice
Attachments:
Untitled 2.vi ‏15 KB

String tokenizer question

Similar Messages

Maybe you are looking for