Inconsistent FileReader behaviour during heavy load

Question

I am using a StreamTokenizer / FileReader to parse the contents of files and have noticed inconsistent results similar to the ones we get during concurrency problems. Only in my case its a single single Thread calling the parser method.
I have run the parser 672 times over the same file and I have a counter that counts the parsed words. Though I would expect the counter to always show the same number, suprisingly it sometimes shows a smaller number !
---1---[227]
---1---[227]
---1---[227]
---1---[227]
---1---[190] <====
---1---[227]
---1---[227]
---1---[227]
---1---[227]
---1---[227]
---1---[227]
---1---[227]
I have the feeling StreamTokenizer.TT_EOL may return EOF prematurely. Then again I need a deterministic way of processing the contents of as many files as needed. How can I make sure my code will always process the same number of tokens for the same file at any given time?
Here the method that produces the inconsistent counts:
    public HashMap<String,ArrayList<Long>> parseDocument( Reader       _reader
                                                        , int          _pivDocID
                                                        , yxStopList   _yxStpLst
                                                        , String       _filename )
            throws   FileNotFoundException
                   , IOException
        HashMap<String,ArrayList<Long>> _postingLists = new HashMap<String,ArrayList<Long>>(HASHMAP_INITIAL_SIZE);
        ArrayList<Long>                 _offsets      = null;
        int                             _currOffset   = 0;
        int                             _numericTokens= 0;
        String                          _token        = null;
        BufferedReader                  _buffReader   = new BufferedReader(_reader);
        StreamTokenizer                 _st           = new StreamTokenizer(_buffReader);
        _st.resetSyntax();
        _st.ordinaryChars(0,255);
        _st.eolIsSignificant(true);
        _st.lowerCaseMode(true);
        _st.whitespaceChars(',', ',' ); // COMMA
        _st.whitespaceChars(' ', ' ' ); // SPACE
        _st.whitespaceChars('.', '.' ); // PERIOD
        _st.whitespaceChars('\t','\t'); // TAB
        _st.whitespaceChars('\n','\n'); // EOL
        _st.whitespaceChars('\r','\r'); // EOL
        _st.wordChars('a','z');
        _st.wordChars('A','Z');
        _st.wordChars('0','9');
        _st.wordChars('_','_');
scan:
        while(true)
            try
                switch(_st.nextToken())
                        case StreamTokenizer.TT_WORD  :
                             _token = _st.sval;
                             if (_token.length() < MINIMUM_ACCEPTABLE_TOKEN_LENGTH) break;
                             if (_token.length() > MAXIMUM_ACCEPTABLE_TOKEN_LENGTH) break;
                             if (_token.matches(".*[^a-zA-Z0-9_].*")) break;
                             if (_token.indexOf("__") > -1) break;
                             if (_token.matches(".*[0-9].*") && _token.matches(".*[a-zA-Z_].*")) break;
                             if (!_token.matches("[a-zA-Z_]+"))
                                 if ( !_token.startsWith("0" ) )               break;    //only numbers like: 069 456456 or 004916099113815
                                 if (  _token.startsWith("000") )              break;
                                 if (    _token.length() == 4
                                      && !(    _token.startsWith("19")
                                            || _token.startsWith("20")
                                            || _token.startsWith("21")) )      break;
                                 if (_token.length() > 20)                     break;
                                 _numericTokens++;
                                 if (_numericTokens > MAXIMUM_NUMBERS_PER_DOC) break;    //do not allow too many numeric tokens per document
                             }//end of [IF]
                             if ( _yxStpLst.isStopWord(_token) )
                                 yxL.log(6,"[yxParser  --  parseDocument(2)]","","WARNING","REJECTING STOPWORD ["+_token+"] !");
                                 break;
                             _currOffset++;
                             if ( !_postingLists.containsKey(_token) )
                                _offsets = new ArrayList<Long>();
                                _offsets.add(0,(long)_pivDocID);
                                _offsets.add(1,(long)_currOffset);
                                try {
                                   _postingLists.put(_token,_offsets);
                                }catch(OutOfMemoryError e001){
                                   e001.printStackTrace();
                                   yxL.log(2,"[yxParser  --  parseDocument(2)]","","ERROR","OutOfMemory while parsing ["+_filename+"]");
                                   break scan;
                             }//end of [IF]
                             else
                                _offsets = _postingLists.get(_token);
                                if (_offsets.size() == MAX_OFFSETS_PER_TOKEN) break;
                                int _prevSumOfOffsets = 0;
                                for ( int i1 = 1;                                                       // ignore i1=0 because i1=0 is the DOCID
                                      i1<_offsets.size();                                               // loop until end of encoded Offsets
                                      i1++ ) _prevSumOfOffsets += _offsets.get(i1);                     // sum all existing encoded offsets
                                _offsets.add(_offsets.size(),(long)(_currOffset - _prevSumOfOffsets));
                                try {
                                   _postingLists.put(_token,_offsets);              // put : replaces existing Key
                                }catch(OutOfMemoryError e001){
                                   e001.printStackTrace();
                                   yxL.log(2,"[yxParser  --  parseDocument(2)]","","ERROR","02 [yxParser] OutOfMemory while parsing ["+_filename+"]");
                                   break scan;
                                yxL.log(6,"[yxParser  --  parseDocument(2)]","","INFO"
                                         ,"02   Updating  ["+_token+"]["+Arrays.toString(_offsets.toArray())+"]");
                             break;
                        case StreamTokenizer.TT_NUMBER: break;                  // Numbers will be treated as Strings
                        case StreamTokenizer.TT_EOL   : break;                  // EOL
                        case StreamTokenizer.TT_EOF   : break scan;             // EOF
                        default                       :
                             break;                                             // individual 1-char tokens will be ignored
                }//end of [SWITCH]
            }catch (Exception e){e.printStackTrace();}
        }//end of [WHILE]
        _buffReader.close();
        _reader.close();
        _token      = null;
        _st         = null;
        _buffReader = null;
        _offsets    = null;
        int _tokensFound = _postingLists.size();
        if (_tokensFound < 1)
            yxL.log(3,"[yxParser  --  parseDocument(2)]","","WARNING","Number of tokens found = ["+_postingLists.size()+"]["+_filename+"]");
            if (sh.isIndexable(_filename))
                _offsets = new ArrayList<Long>();
                _offsets.add(0,(long)_pivDocID);
                _offsets.add(1,1L);
                try {
                   _postingLists.put(sh.MANUAL_INDEX_FILE_START,_offsets);
                }catch(OutOfMemoryError e001){
                   e001.printStackTrace();
                   yxL.log(2,"[yxParser  --  parseDocument(2)]","","ERROR","OutOfMemory2 while parsing ["+_filename+"]");
            }//end of [IF]
        else
            yxL.log(4,"[yxParser  --  parseDocument(2)]","","INFO","Number of tokens found = ["+_postingLists.size()+"]");
        COUNT_TOTAL_WORDS += _postingLists.size();
System.out.println("---1---["+_postingLists.size()+"]");
        yxL.log(6,"[yxParser  --  parseDocument(2)]","END");
        return _postingLists;
    }

Answer 1

Yes. there is a GUI involved. The Gui participated in this only by triggering the parsing process through the click of a button. The parser receives its input from a Vector which contains fully qualified filenames.
Each filename is used to instantiate a FileReader Object. This in turn is passed on to the parser method shown above. The inconsistency is located in the method above. I have already searched for days to reach to this conclusion. I was taking for granded that the stream would be read till EOF is reached, but this is not always the case. The method shown above is re-run over 600 times and 2% of the time it return 10-20 words less. !!