Unable to read 2 GB file??

Question

I want to read a text file of size Max upto 200 GB.
I tried with BufferedReader -- > it give error "out of memory " the moment i call readLine() method.
I tried to read stream character by character but still give error once file size increaese to 150MB.
I tried to broke the file and try to read the file But the again after reading the 4 file each of 40 MB it again broke "Out Of memory"
I am totally stuck.
I dont know how to go ahead can any one help me out in same ?
Note i need to maintain the count of URLs in the text file ??
Please help me out.

Answer 1

Hi JoachimSauer ,
Thanks for your reply..
My task is to read the file and extract URLs and the count the repetitions.
The file which i am using is can be 1 GB.
yes it has newline charachter.
Actually when i tried to read char by char i made a method which returning me the line ::
please see readLine method
public class FileChunkReading {
private FileInputStream in;
private StringBuffer buf = new StringBuffer();
public FileChunkReading(String fileName) throws FileNotFoundException {
in = new FileInputStream(fileName);
{color:#0000ff}public String readLine() throws IOException {
int c;
buf.setLength(0);
buf.setLength(1024);
c = in.read();
if(c == -1){
return null;
while (c != -1) {
if ((char)c == '\n')
return buf.toString();
else
buf.append((char) c);
c = in.read();
return buf.toString();
}{color}
public void closeFile() throws IOException {
in.close();
//// Now this read line method is used to reading one line at a time.
Now i had made a class which is using instance of this class.
when it run after few minutes give out memory exception !
The other class which is using it is as follows :
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
class FileURLSearch {
     private FileChunkReading fcr = null;
     private String searchTypes[] = {".jsp", ".htm" ,".html",".pl"};
     private HashMap <String,Integer>urlMap = new HashMap<String,Integer>();
     public FileURLSearch(String fileName) throws FileNotFoundException {
          fcr = new FileChunkReading(fileName);
     public Map getURLList() throws IOException {
               String currentLine = null;
               while((currentLine = fcr.readLine()) != null) {
                    String url = extractURL(currentLine );
                    if(url != null && url !="") {
                         String urls[] = url.split(";");
                         for(int i = 0 ; i < urls.length ; i++){
                              if(urlMap.containsKey(urls)){
                                   urlMap.put(urls[i], (Integer)urlMap.get(urls[i]) + 1);
                              }else{
                                   urlMap.put(urls[i], 1);
          return urlMap;
     //it might be possible the current url is extended till previous line
     private String extractURL(String currentLine ) {
          StringBuffer sb = new StringBuffer();
          int startIndex = 0;
          for(String st : searchTypes){
               while( (startIndex = currentLine.indexOf(st , startIndex))!=-1){
                    //backtrack
                    int urlStartIndex = backTrackURL(currentLine , startIndex);
                    startIndex += 3;
                    int urlEndIndex = startIndex;
                    //forwardtrack
                    if(currentLine.charAt(++startIndex) == '?'){
                         urlEndIndex = forwardTracking(currentLine ,startIndex);
                    sb.append(currentLine.substring(urlStartIndex, urlEndIndex));
                    sb.append(";");
          return sb.toString();
     // return end index
     private int forwardTracking(String currentLine ,int startIndex){
          return minValue(currentLine.indexOf(';',startIndex) ,currentLine.indexOf(' ',startIndex) ,currentLine.indexOf('"',startIndex))+1;
     //return start index
     private int backTrackURL(String currentLine ,int endIndex) {
          return maxValue(currentLine.lastIndexOf(';',endIndex) ,currentLine.lastIndexOf(' ',endIndex) ,currentLine.lastIndexOf('"',endIndex))+1;
     private int maxValue(int val1 , int val2 , int val3) {
          return (val1 > val2) ? ((val1 > val3)? val1 : val3 ) : ((val2 > val3) ? val2 : val3);
     private int minValue(int val1 , int val2 , int val3) {
          return (val1 < val2) ? ((val1 < val3)? val1 : val3 ) : ((val2 < val3) ? val2 : val3);
public class FileReaderDemo {
     public static void main(String[] args) throws IOException {
          FileURLSearch fus = null;
          try {//commmand line argument sort of
               fus = new FileURLSearch("testLogFile.log");
          } catch (FileNotFoundException e) {
               e.printStackTrace();
          long startTime = System.currentTimeMillis();
          Map map =fus.getURLList();
          System.out.println(" URL Found : "+map.size());
          System.out.println(" URL Found : "+map);
          System.out.println("time taken : " + ((System.currentTimeMillis() - startTime)/1000) +" seconds" );
Please suggest where i am going wrong .
Thanks in Advance.
Edited by: zuned on Jul 22, 2010 5:16 AM