Problems with String.split(regex)

Question

Hi! I'm reading from a text file. I do it like this: i read lines in a loop, then I split each line into words, and in a for loop I process ale words. Heres the code:
BufferedReader reader = new BufferedReader(new FileReader(file));
String line;
while ((line = reader.readLine()) != null) {
    String[] tokens = line.split(delimiters);
    for (String key : tokens) {
doSthToToken();
reader.close();The problem is that if it reads an empty line, such as where is only an "\n", the tokens table has the length == 1, and it processes a string == "". I think the problem is that my regex delimiters is wrong:
String delimiters = "\\p{Space}|\\p{Punct}";
Could anybody tell me what to do?

Answer 1

Ok, so what do you suggest?I suggest you don't worry about it.
Or if you are worried then you need to test the two different solutions and do some timings yourself.
And how do you know the regex lib is so slow and badly written?First of all slowness is all relative. If something takes 1 millisecond vs 4 milliseconds is the user going to notice? Of course not which is why you are wasting your time trying to optimize an equals() method.
A general rule is that any code that is written to be extremely flexible will also be slower than any code that is written for a specific function. Regex are used for complex pattern matching. StringTokenizer was written specifically to split a string based on given delimiters. I must admit I haven't tested both in your exact scenario, but I have tested it in other simple scenarios which is where I got my number.
By the way I was able to write my own "SimpleTokenizer" which was about 30% faster than the StringTokenizer because I was able to make some assumptions. For example I only allowed a single delimiter to be specified vs multiple delimiter handled by the StringTokenizer. Therefore my code could be very specific and efficient. Now think about the code for a complex Regex and how general it must be.