Trying to read in a hell a lot of strings -- 657603

Question

Hi-ya
I am trying tto read in the DMOZ structure data, but no matter what I get an out of memory error. I have set max on the jvm plus I am not constructing anything unncessary except for the strings. The file size is only 52meg so why should java not be able to handle it?
First I tried bufferedreader and had no luck, so i thought this baby would do it. What am I doing silly? Thanks.
This is basicly the algorithm
int noofbytes = 31607568
for (int i = 0; i < noofbytes ; i++)
if (bytes[i] != 0x0A)
offset++;
else
byte[] theline = getBytes(bytes, (i - offset), i);
offset = 0;
String temp = new String(theline, "UTF-8");
counter++;
strings[counter] = temp;
temp = null;
theline = null;
//System.out.println(temp);
static public byte[] getBytes(byte[] bytes, int start, int stop)
int len = (stop - start);
byte[] newArray = new byte[len];
for (int i = 0; i < len; i++)
newArray[i] = bytes[start + i];
return newArray;
}

Answer 1

Try the following with the -Xmx256m I suggested. Note you cannot set the -Xmx to higher than the JVM supports.
import java.util.ArrayList;
import java.util.List;
import java.io.*;
public class TestLargeTextFile {
    private static final String TEST_FILE = "deleteme.txt";
    public static void generateLargeTextFile(String fileName, int lines, int length) throws IOException {
        StringBuffer sb = new StringBuffer();
        for(int j=0;j<length;j++)
            sb.append((char) (j % 64 + ' '));
        String line = sb.toString();
        PrintWriter printWriter = new PrintWriter(new BufferedWriter(new FileWriter(fileName)));
        for(int j=0;j<lines;j++)
            printWriter.println(line);
        printWriter.close();
        File file = new File(fileName);
        System.out.println("Created file "+fileName+" with "+lines+" size="+file.length()+" bytes");
    public static List readFile(String fileName) throws IOException {
        BufferedReader br = new BufferedReader(new FileReader(fileName));
        List ret = new ArrayList();
        String line = null;
        while((line = br.readLine()) != null)
            ret.add(line);
        return ret;
    public static void main(String[] args) throws IOException {
        try {
            generateLargeTextFile(TEST_FILE, 657603, 80);
            List list = readFile(TEST_FILE);
            System.out.println("Read "+TEST_FILE+" with "+list.size()+" lines.");
        } finally {
            new File(TEST_FILE).delete();
}Prints
Created file deleteme.txt with 657603 size=53923446 bytes
Read deleteme.txt with 657603 lines.