StreamTokenizer

I'm implementing a program that reads some variable values representing a day and an hour (ex. monday 10am = m10, sunday 3pm = s15)
Then I need to import to a database the day and the hour. I was wondering how can I use the StreamTokenizer class to separate the numbers from the letters, or if there's a better way to do it. Please, don't send to the API, I have been there all day and still don't got it, thanks

do you mean StringTokenizer?
What don't you get?! You create a tokenizer object that looks for certain characters (or maybe just one) and counts the number of substrings between instances of them... the number of substrings are 'tokens' and the splitting characters are called delimeters.
Hopefully that will clear up the terminology in the API...
However, you may as well just use String.substring() method providing the format will always remain as specified.

Similar Messages

How can i read a string with nextToken() of StreamTokenizer

I need for my class paper to read a string from a file and i used the StreamTokenizer's method nextToken but i can not read a string with it. my code is:
StreamTokenizer st = new StreamTokenizer(the code that gets the input from the file)
String line;
while ((line !=br.readLine()) != null) {
String surname = (st.nextToken()).trim();
but it gets me some error of:
int can not be dereferenced
what should I do to get the string i need?

Look at the API for java.io.StreamTokenizer. In particular, look at the return type for nextToken().
Good luck.

StreamTokenizer NVAL bug? 16 digits not reading correctly?

hi all!
i do some calculations which are highly precise, and we use numbers with 16 digits after the comma. this works most of the time without problems. but for some numbers java rounds it automatically. i think the problem is within the StreamTokenizer nval method.
but please see the code to get what i mean:
private static void readFromFileTestNumberProblem() {
          try{
               FileReader reader = new FileReader(new File("test.txt"));
               StreamTokenizer tokenStream = new StreamTokenizer(reader);
               //recognize end of lines as separated tokens. ttype will be set to TT_EOL
               tokenStream.eolIsSignificant(true);
               tokenStream.nextToken(); // get the first token
               String flag = (String) tokenStream.sval;
               while(flag.equals("START")){
                    System.out.println("------------------------");
                    for (int i = 0; i < 5; i++) {
                         if(tokenStream.nextToken() == tokenStream.TT_NUMBER){
                              System.out.println((Double)tokenStream.nval);
                         else throw new Exception("Check Input File - Not " + nObjectives + " Objectives in line " + tokenStream.lineno());
                    // check for END token...
                    if(tokenStream.nextToken() == tokenStream.TT_WORD){
                         if (tokenStream.sval.equals("END")) {
                              flag = (String)tokenStream.sval;
          catch (Exception e) {
               e.printStackTrace();
     }and a file to test it: save it into test.txt
START 7106199.0 5.0376 0.5068132912149117 0.8624999999999999 225450.0
7595899.0 4.764800000000001 0.6378176973051516 0.9624999999999999 281000.0 END
result is:
7106199.0
5.0376
0.5068132912149117
0.8624999999999999
225450.0
7595899.0
4.764800000000001
0.6378176973051516
0.9625 --> i dont like this value :-) it should be 0.9624999999999999
281000.0
is this a bug in nval? or is this normal?
if it is normal, can someone tell me how can i get around this...
would appreachiate any help ..
thanks
thomas

that does not change anything..
private static void readFromFileTestNumberProblem() {
          try{
               FileReader reader = new FileReader(new File("test.txt"));
               StreamTokenizer tokenStream = new StreamTokenizer(reader);
               //recognize end of lines as separated tokens. ttype will be set to TT_EOL
               tokenStream.eolIsSignificant(true);
               tokenStream.nextToken(); // get the first token
               String flag = (String) tokenStream.sval;
               tokenStream.wordChars('0', '9');
               while(flag.equals("START")){
                    System.out.println("------------------------");
                    for (int i = 0; i < 5; i++) {
                         tokenStream.nextToken();
                         System.out.println(tokenStream.toString());
                    // check for END token...
                    if(tokenStream.nextToken() == tokenStream.TT_WORD){
                         if (tokenStream.sval.equals("END")) {
                              flag = (String)tokenStream.sval;
          catch (Exception e) {
               e.printStackTrace();
     }same result as before..
7106199.0
5.0376
0.5068132912149117
0.8624999999999999
225450.0
7595899.0
4.764800000000001
0.6378176973051516
0.9625
281000.0

What alternative wuld there be to using StreamTokenizer?

Hello,
Currently I am using a StreamTokenizer to parse data from a file. The problem is that it goes way too slow. I'd like a faster way to process the data. If there is a better alternative to using StreamTokenizer I'd like to know. I've looked around and haven't found anything. I though using the java.nio stuff might help but now the I/O is faster in comparison to the BufferedReader, but the StreamTokenizer is keeping the I/O capabilities slow. The file sizes range from 9M - 20M and I need to find values and hold the data. I listed a very generic structure for the files I'm reading in below. I use the Headings(ex: [Values]) to sort through the data under the heading and then move on using the token.nextToken. If anyone could give me some ideas I'd appreciate it. Thank you in advance.
An Example:
[Values]
1 = "String1"
2 = "String2"
[Data Part1]
JOB_1 = 1021201212
PART_1 = 21231331
[TESTS]
T_000_= Data; Data;Data
through
T_1442 = Data; Data; Data

Currently I am using a StreamTokenizer to parsedata
from a file.Write a custom parser.
And if you parsing the same data more than once then
put the data into another format on the first parse so
it is faster on the subsequent tries.I think this would be your best bet right now. Because public static void mySimpleTokenizer()String s, String delimiter)
   String sub = null;
   int i =0;
   int j =s.indexOf(delimiter); // First substring
   while( j >= 0) {
   sub = s.substring(i,j);
     i = j + 1;
     j = s.indexOf(delimiter, i);   // Rest of substrings
   sub = s.substring(i); // Last substring
}I read some where that the above method works almost 4 times faster than StringTokenizer, because of less overhead.
Also just wondering if the following links could be of any help
http://ostermiller.org/utils/StringTokenizer.html
http://www.javaperformancetuning.com/news/roundup032.shtml
http://www.ftponline.com/javapro/2002_08/online/servletsjsp_08_06_02/Java%20Servlets%20Ch16.pdf

Can i link a StreamTokenizer with a String Tokenizer?

As recently informed, I now know that I can treat delimiters as tokens using StringTokenizer. My next question, is how do I use a StringTokenizer with a StreamTokenizer to let me read a word (eg cat) as a token and also a "," as a token? The reason being that I would like to be able to use the 'end of file' and 'end of line' flags in the StreamTokenizer but also would like to treat certain delimiters as tokens using the StringTokenizer!
If this is not possible, would I need to use the StringTokenizer and treat a space(" ") as a token (or delimiter)?
If I read a token with the StringTokenizer can I expect the StreamTokenizer to read the next token after that, or is that wishfull thinking?
The following code was used just to read words as tokens. I would like to incorporate the StringTokenizer to set delimiters as tokens to be read. How??????
try
    File file = new File("test.txt");
    FileInputStream stream = new FileInputStream (file);
    InputStreamReader reader = new InputStreamReader (stream);
    BufferedReader buffer = new BufferedReader (reader);
    StreamTokenizer tokenize = new StreamTokenizer (buffer);
    tokenize.nextToken();
    while (tokenize.ttype != StreamTokenizer.TT_EOF)
        data = tokenize.sval;
        tokenize.nextToken();
catch(IOException e)
    System.out.println("Error in file input:" + e.toString());
    }

I don't think you should do it that way, it can get a little nasty. Just use a BufferedReader to read one line in at a time, then use a StringTokenizer to break it up. When a reader has reached the eof, the readLine returns a null.

ToString and StreamTokenizer - general use questions

Hello everyone.
I am currently taking a class in Java at a NYC school, and got into an argument with the teacher regarding the proper use of the toString method and the SteamTokenizer class. He claims the following:
1. toString() methods are primarily for debugging - i.e., I defined this method in a class I created in order to better represent the class' state visually - and he marked that as wrong - since I was not using the method 'properly'.
2. I was using the StreamTokenizer class in order to read in user input from the terminal - and only perform something if values entered were numbers. He claims that my use of the class is inappropriate - since it "doesn't do what I think it does" - whatever that means.
I just wanted to see what people's feelings were on the proper use of the toString method and the StreamTokenizer class (in regards to reading user input). Thanks!

also on your last remark,
if the example you use the toString-method insteadof
the getAccountNumber-method: how is a developer to
know what the toString-method will give; that'swhat
I mean with vague. 'getAccountNumber' does notleave
much roam for imagination.Yes, but if you give the developer an API for your
program (if he can't see the code himself), the API
will (should!) tell him what toString will return.Maybe you work in a project where developers actually have time to do this. I agree that they should and that the API should always be up-to-date but I've never encountered a project where this is actually done.
Actually, I don't really understand your problem with this statement. Don't you agree that a method getAccountNumber should be created and that this should not only be returned by the toString-method. If you want your toString to return this value you can allways do this, for debugging purposes.
Another important argument; in these cases you do not excpect the toString to return important information that is called from other methods. What if another developer needs to debug and changes the debug-information? I know that where I work it's the first developer who would get blamed for this.

About parseNumbers() method in StreamTokenizer

This is about StreamTokenizer class in java.io package...
If we read two strings..
[1]
say "abcd123"
then sval="abcd123"
But
[2]
second string say
"123abcd"
then for first token we have nval=123.0
and for next token we have sval="abcd"
here ,Output does not change whether we use or don't use parseNumber()
Then when parseNumber() is actually used?
And can anybody give me code which shows significance of parseNumbers() method

If I have understood you say that I can define
abstract a class even if it hasn't abstract methods.
Ok?Yes. Think OO: does it make sense to have a classes "Intern", "Contractor", "Employee", "Client", "Manager"? Yepp.
Does it make sense to make them extend "Person"? Yepp, why not? With address and such.
Does it make sense to have an object that's "just" a person? Not really - whatever this person does, he always has some kind of special semantics and abilities, which means that it might make sense to leave Person abstract - and there aren't even a methods defined yet that would enforce this decision anyway.
An other question: can I imagine that methods in
class ServletInputstream are native as, for example,
they strongly depend on operating system?You wouldn't know if they're not public - at least the API doesn't mention any. I doubt it has native methods itself, but it will probably rely on native methods it inherits from InputStream.
I would
have checked but is not so easy to find source code
for a real server implementation of JEE.
Thanks.Tomcat, JBoss..

Using a BufferedReaderobject in a StreamTokenizer

I can't seem to figure out how to use the BufferedReader object I have created (called input) as the source for a new StreamTokenizer.
I do this
StreamTokenizer adder = new StreamTokenizer(System.setIn(input));
and it says variable input not found in class yet I have created before thus:
try {
FileReader r = new FileReader("c:\\JavaProjects\\TMA01\\Streams\\MyInputFile.txt");
BufferedReader input = new BufferedReader(r);
catch (FileNotFoundException e) {System.exit(0);}
Any ideas anyone.

I think the problem is simply thay 'input' only hac scope within the try block.
This removes the error:
BufferedReader input;
try
FileReader r = new FileReader("c:\\JavaProjects\\TMA01\\Streams\\MyInputFile.txt");
input = new BufferedReader(r);
catch (FileNotFoundException e)
System.exit(0);
however,
StreamTokenizer adder = new StreamTokenizer(System.setIn(input));
doesn't work, as input is not an input stream. I would use FileInputStream

How to use StreamTokenizer class ?

i want to use StreamTokenizer class in my program so that when i read from a file i want to make tokens of words present in the file, but i dont know how to use this class in my program, as i didnt find any example which shows how to use StreamTokenizer class. Can nebody knows how to use it in a program ?

Hi,
here's some sample code of how to use the StreamTokenizer.
in = new StreamTokenizer(
new BufferedReader(
new FileReader(fname)));
You can iterate through the file with:
while(in.nextToken() != StreamTokenizer.TT_EOF)
when you call nextToken() ttype is filled with the type of token, you can't test ttype for what kind of token,
test for word:
if (in.ttype == StreamTokenizer.TT_WORD)
test for word:
if (in.ttype == StreamTokenizer.TT_WORD)
I think you can grasp the rest in javadoc

Replacing letters through StreamTokenizer

Hi,
I am writing a program which reads from a text file, passes it into a StreamTokenizer, looks for a particular word and capitalises the first letter (in this case "apples" to "Apples"). It's all fine, except when the word I'm looking for appears in the middle of a sentence, in which case, it seems to ignore it completely. I'm probably doing something really stupid, but I'd appreciate it if anyone could help me!
Input file:
John likes apples. apples are healthy.
The apples John eats are red, sometimes green. Most
apples come from farms.
Current output file:
John likes apples. Apples are healthy.
The apples John eats are red, sometimes green. Most
apples come from farms.
What output file is supposed to be:
John likes Apples. Apples are healthy.
The Apples John eats are red, sometimes green. Most
Apples come from farms.
Code:
//Filename:FileReader.java
import java.io.*;
public class FileReader
     public static void main(final String[]Args) throws IOException, FileNotFoundException
          //declares and instantiates the array aFileNames
          String[] aFileNames = new String[3];
          //read in the input & output file names and places them into an array
          for(int tArgNames = 0; tArgNames < Args.length; tArgNames ++)
               aFileNames[tArgNames] = Args[tArgNames];
          //instantiates a buffered reader
          BufferedReader tTextIn = new BufferedReader(new FileReader(aFileNames[0]));
          //declares and instantiates a stream tokenizer
          StreamTokenizer tStreamTokens = new StreamTokenizer(tTextIn);
          //declares a print writer for output file
          PrintWriter tTextOut = null;
          //instantiates the print writer for output file
          tTextOut = new PrintWriter(new FileWriter(aFileNames[1]));
          //declares a print writer for log file
          PrintWriter tTextLog = null;
          //instantiates the print writer for log file
          tTextLog = new PrintWriter(new FileWriter(aFileNames[2]));
          //Writes name of files being opened to log file
          tTextLog.println("File being opened: " + aFileNames[0]);
          tTextLog.println("File being opened: " + aFileNames[1]);
          //Line counter variable
          int tWordCounter = 0;
          //Word counter variable set at -1 to account for eof
          int tLineCounter = -1;
          //Recognises punctuation
          tStreamTokens.wordChars('.', '.');
          tStreamTokens.wordChars(',', ',');
          tStreamTokens.wordChars('\'', '\'');
          tStreamTokens.eolIsSignificant(true);
          //while there are more lines of text in the file
          while(tStreamTokens.nextToken() != StreamTokenizer.TT_EOF)
               //if the stream token is a string
               if(tStreamTokens.ttype == StreamTokenizer.TT_WORD)
                    //and if the token value is "apples"
                    if(tStreamTokens.sval.equals("apples"))
                         //change the stream token to "Apples"
                         tStreamTokens.sval = ("Apples");
                         tTextLog.println("Word changed at Line " + tStreamTokens.lineno());
                    //increment the word counter
                    tWordCounter++;
               //if the end of the line has been reached, increment the line counter
               else if(tStreamTokens.ttype == StreamTokenizer.TT_EOL)
                    tStreamTokens.sval = ("");
                    tTextOut.println("");
                    tLineCounter ++;
                    tTextLog.print("End of line reached. Word Count: " + tWordCounter + " . ");
                    tTextLog.println("Line Count: " + tLineCounter);
               //writes the current token to the output file named in the command line
               tTextOut.print(tStreamTokens.sval + " ");
          //prints the values to the designated output file
          tTextOut.println("Name of input file: " + aFileNames[0]);
          tTextOut.println("Number of lines: " + tLineCounter);
          tTextOut.println("Number of words: " + tWordCounter);
          //prints the values to the command prompt
          System.out.println("Name of input file: " + aFileNames[0]);
          System.out.println("Number of lines: " + tLineCounter);
          System.out.println("Number of words: " + tWordCounter);
          //prints the values to the log file
          tTextLog.println("Total Number of Lines: " + tLineCounter);
          tTextLog.println("Total Number of Words: " + tWordCounter);
          //Writes name of files being closed to log file
          tTextLog.println("File being closed: " + aFileNames[0]);
          tTextLog.println("File being closed: " + aFileNames[1]);
          //Closes the files
          tTextIn.close();
          tTextOut.close();
          tTextLog.close();
}Apologies for the not-very-nice code. I'd be grateful for any help on this.
C

for one, you might want it to check for "apples."Thanks for your help! I thought I was already
checking, here:
//and if the token value is "apples"
     if(tStreamTokens.sval.equals("apples")) ///<--here
          //change the stream token to "Apples"
          tStreamTokens.sval = ("Apples");
tTextLog.println("Word changed at Line " +
+ tStreamTokens.lineno());
     }but I'll have to look into it some more.
C
maybe you should read the post again. you need to check for the period too, dork.
     if(tStreamTokens.sval.equals("apples")) {
     //change the stream token to "Apples"
             tStreamTokens.sval = ("Apples");
          tTextLog.println("Word changed at Line " + tStreamTokens.lineno());
     }should be something like
     if(tStreamTokens.sval.equals("apples")) {
     //change the stream token to "Apples"
             tStreamTokens.sval = ("Apples");
          tTextLog.println("Word changed at Line " + tStreamTokens.lineno());
        }else if(tStreamTokens.sval.equals("apples.")) {
     //checking for "apples." also
             tStreamTokens.sval = ("Apples.");
          tTextLog.println("Word changed at Line " + tStreamTokens.lineno());

StreamTokenizer or any other alternative?

I am looking for a way to read a file that contains words and decimals separated by ;
The file looks something like this:
String ; Decimal
String ; Decimal
String ; Decimal
I think that StreamTokenizer could be a way to do it, but can't find any info on it.
Any ideas on where to find the info or other alternative to play around with?
Thanks!!!

I thought StringTokenizers were hard, but that's not the case. I think it would be best to use a StringTokenizer in this case, so here's how I would do it...import java.util.*;
class Token
    public Token ()
        String str = new String ("String ; Decimal");
        // A regualr string with things we want to "filter" out.
        StringTokenizer st = new StringTokenizer (str, "; ");
        // The delimeters are ';' and ' ' (a space). So when
        // the string is printed, you get just the words, nothing
        // else. This can easily be adapted and re-arranged, so play around
        // with it and try different things.
        while (st.hasMoreTokens ())
            System.out.println (st.nextToken ());
        // The part above has to iterate over the sentence and collect tokens
        // to print. While it does have more to print, it prints each one.
    public static void main (String [] args)
        new Token ();
}And try this link: http://java.sun.com/j2se/1.4/docs/api/index.html it has info about the different classes in Java, the StringTokenizer is in java.util.StringTokenizer; or java.util in short.
Check it out!

StreamTokenizer static fields

Maybe this is a stupid question but why does this compile without problems:
Reader r = new BufferedReader(new InputStreamReader(System.in));
StreamTokenizer tok = new StreamTokenizer(r);
          // StreamTokenizer tok = new StreamTokenizer(System.in);
          try{
               while (tok.nextToken() != StreamTokenizer.TT_EOF){
                    switch (tok.ttype) {        // ttype is token type
                         case StreamTokenizer.TT_NUMBER:
                              System.out.println("number " + tok.nval);
                              break;
     ....But this:
Reader r = new BufferedReader(new InputStreamReader(System.in));
StreamTokenizer tok = new StreamTokenizer(r);
          // StreamTokenizer tok = new StreamTokenizer(System.in);
          try{
               while (tok.nextToken() != StreamTokenizer.TT_EOF){
                    switch (tok.ttype) {
                         case tok.TT_NUMBER: // nval is numeric value
                              System.out.println("number " + tok.nval);
                              break;
     ....                    gives the following compiling error:
constant expression required
    case tok.TT_NUMBER: value
           ^
1 errorI thought static fields can be access both through the name of the class and objects of it.
The version that doesn't compile comes from a book which makes it slightly more confusing.

excuse my wording. I have been using >= mostly, though
I do know that > exists. If I replace the >
with >= and compile there are no errors. But I also
get the following errors with the >
# javac TestLight.java
TestLight.java:13: ')' expected
if (a.KWH_PRICE > 0)
^
TestLight.java:13: not a statement
if (a.KWH_PRICE > 0)
^
TestLight.java:13: ';' expected
if (a.KWH_PRICE > 0)
^
TestLight.java:13: cannot resolve symbol
symbol : variable gt
location: class TestLight
if (a.KWH_PRICE > 0)
^
4 errorsYou really get errors then ?
Very strange, I don't get any.....

About streamtokenizer

I want to write a program, which will place filenames inside binary tree nodes. I will then be able to do a fast search.
I work with "StreamTokenizer" and i have some questions and problems:
How is token defined? Is it the all the characters contained inside 2 space characters, ie. a word? Is it a single character (letter, number, special character)?
Can i configure a StreamTokenizer object to consider a single character/a word as a token?
I have read the API, but the output of my program is confusing.
The /home/kostas/tmp is a file containing an "ls -laR" output. Here's exactly a part of what i get:
3919.0
Jun
18.0
2.0
34.0
f0
84.0
rw-r--r--
1.0
root
root
92070.0
Jun
18.0
2.0
34.0
f
85.0
rw-r--r--
1.0
root
root
1207210.0
Jun
18.0
2.0
34.0
f0
85.0
rw-r--r--
1.0
root
Sometimes the size of the file, sometimes the permissions, no filename, and plenty of "newlines".
Here's the code:
import java.io.*;
import java.util.*;
class FastFileFind
    private static void analyzeInputCreateNodes() throws IOException
        File f= new File("/home/kostas/tmp");
        FileInputStream fin= new FileInputStream(f);
        DataInputStream in= new DataInputStream(fin);
        StreamTokenizer tokenizer=new StreamTokenizer(in);
        tokenizer.ordinaryChar(' ');
        String s="";
        while(tokenizer.nextToken() !=StreamTokenizer.TT_EOF)
            switch(tokenizer.ttype)
            case StreamTokenizer.TT_EOL:
                s = new String("EOL");
                break;
              case StreamTokenizer.TT_NUMBER:
                s = Double.toString(tokenizer.nval);
                break;
            case StreamTokenizer.TT_WORD:
                s = tokenizer.sval; // Already a String
                break;
            default: // single character in ttype
                s = String.valueOf((char)tokenizer.ttype);
            System.out.println(s);
    public static void main(String args[]) throws IOException
        analyzeInputCreateNodes();
}

resetSyntax() fixed all the problems.

StreamTokenizer code: Don't understand.

Can anyone explain in dumbass language what is going on here? I mean, I know the end result, but I don't exactly understand what is going on. Little help?
char separator = ',';
StreamTokenizer tokenizer = new StreamTokenizer(new BufferedReader(new StringReader(str)));
tokenizer.resetSyntax();
tokenizer.wordChars('\u0000', (char)(separator -1)); // Everything is a word character.
tokenizer.wordChars((char)(separator + 1), '\u00ff'); // except for the separator.The first two lines are just there to give you a sense of context. :) My main problem is the three calls to the tokenizer methods. What on earth is going on there?
Thanks for any help. :)

char separator = ',';
StreamTokenizer tokenizer = new StreamTokenizer(new
BufferedReader(new StringReader(str)));
tokenizer.resetSyntax();All characters are "regular" for this tokenizer--none are considered "word" or "number" or "whitespace" characters.
tokenizer.wordChars('\u0000', (char)(separator -1));
// Everything is a word character. Everything from character 0 up to, but not including, the comma character is considered a "word" character.
tokenizer.wordChars((char)(separator + 1), '\u00ff');
// except for the separator.Everything after the comma, up to character 0x00ff is also a "word" character.The docs for StreamTokenizer say it can be used to break a stream up into words, numbers, comments, etc. We've defined that a comma is "ordindary" and everything else (up to char 255) is part of a word.

Reading xml with streamtokenizer

Does anyone know how to read xml tags (everthing in between < & >) with streamtokenizer, I can't find any tutorials.

I recommend xml packages. It's a bit more complex, IMO.

How to use StreamTokenizer

hello
i have a file like that :
02-20-04 11:02AM 1307472 liste7.pdf
02-20-04 11:02AM 1807972 liste2.pdf
02-20-04 11:02AM 1557972 liste1.pdf
i want to use StreamTokenizer to get the number part 1307472 ,1807972 , 1557972
knowing that they bigin all at coolumn 31 and finich at column 38
then i want to convet it into an int
and to add them
hellllllllp

do you know how i can convert a setring to an integerThe Integer class has a parseInt method. You must enclose it in a try block and catch the NumberFormatException it can throw if the String isn't a valid int.

StreamTokenizer

Similar Messages

Maybe you are looking for