Out of memory error - from parsing a "fixed width file"

Question

This may be fairly simple for someone out there but I am trying to write a simple program that can go through a "fixed width" flat txt file and parse it to be comma dilmeted.
I use a xml file with data dictionary specifications to do the work. I do this because there are over 430 fields that need to be parsed from a fixed width with close to 250,000 lines I can read the xml file fine to get the width dimensions but when I try to apply the parsing instructions, I get an out of memory error.
I am hoping it is an error with code and not the large files. If it is the latter, does anyone out there know some techniques for getting at this data?
Here is the code
   import java.io.*;
   import org.w3c.dom.Document;
   import org.w3c.dom.*;
   import javax.xml.parsers.DocumentBuilderFactory;
   import javax.xml.parsers.DocumentBuilder;
   import org.xml.sax.SAXException;
   import org.xml.sax.SAXParseException;
    public class FixedWidthConverter{
      String[] fieldNameArray;
      String[] fieldTypeArray;
      String[] fieldSizeArray;      
       public static void main(String args []){
         FixedWidthConverter fwc = new FixedWidthConverter();
         fwc.go();
         fwc.loadFixedWidthFile();
        //System.exit (0);
      }//end of main
       public void go(){
         try {
            DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();
            DocumentBuilder docBuilder = docBuilderFactory.newDocumentBuilder();
            Document doc = docBuilder.parse (new File("files/dic.xml"));
            // normalize text representation            doc.getDocumentElement ().normalize ();
            System.out.println ("Root element of the doc is " +
                 doc.getDocumentElement().getNodeName());
            NodeList listOfFields = doc.getElementsByTagName("FIELD");
            int totalFields = listOfFields.getLength();
            System.out.println("Total no of fields : " + totalFields);
            String[] fldNameArray = new String[totalFields];
            String[] fldTypeArray = new String[totalFields];
            String[] fldSizeArray = new String[totalFields];
            for(int s=0; s<listOfFields.getLength() ; s++){
               Node firstFieldNode = listOfFields.item(s);
               if(firstFieldNode.getNodeType() == Node.ELEMENT_NODE){
                  Element firstFieldElement = (Element)firstFieldNode;
                  NodeList firstFieldNMList = firstFieldElement.getElementsByTagName("FIELD_NM");
                  Element firstFieldNMElement = (Element)firstFieldNMList.item(0);
                  NodeList textFNList = firstFieldNMElement.getChildNodes();
                  //System.out.println("Field Name : " +
                           //((Node)textFNList.item(0)).getNodeValue().trim());
                  //loads values into an array
                  //fldNameArray[s] = ((Node)textFNList.item(0)).getNodeValue().trim();
                  NodeList typeList = firstFieldElement.getElementsByTagName("TYPE");
                  Element typeElement = (Element)typeList.item(0);
                  NodeList textTypList = typeElement.getChildNodes();
                  //System.out.println("Field Type : " +
                           //((Node)textTypList.item(0)).getNodeValue().trim());
                  //loads values into an array
                  //fldTypeArray[s] = ((Node)textTypList.item(0)).getNodeValue().trim(); 
                  NodeList sizeList = firstFieldElement.getElementsByTagName("SIZE");
                  Element sizeElement = (Element)sizeList.item(0);
                  NodeList textSizeList = sizeElement.getChildNodes();
                  //System.out.println("Field Size : " +
                           //((Node)textSizeList.item(0)).getNodeValue().trim());
                  //loads values into an array
                  fldSizeArray[s] = ((Node)textSizeList.item(0)).getNodeValue().trim();   
               }//end of if clause
            }//end of for loop with s var
            //setFldNameArray(fldNameArray);
            //setFldTypeArray(fldTypeArray);
            setFldSizeArray(fldSizeArray);
             catch (SAXParseException err) {
               System.out.println ("** Parsing error" + ", line "
                  + err.getLineNumber () + ", uri " + err.getSystemId ());
               System.out.println(" " + err.getMessage ());
             catch (SAXException e) {
               Exception x = e.getException ();
               ((x == null) ? e : x).printStackTrace ();
             catch (Throwable t) {
               t.printStackTrace ();
      }//end go();
       public void setFldNameArray(String[] s){
         fieldNameArray = s;
      }//end setFldNameArray
       public void setFldTypeArray(String[] s){
         fieldTypeArray = s;
      }//end setFldTypeArray
       public void setFldSizeArray(String[] s){
         fieldSizeArray = s;
      }//end setFldSizeArray
       public String[] getFldNameArray(){
         return fieldNameArray;
      }//end setFldNameArray
       public String[] getFldTypeArray(){
         return fieldTypeArray;
      }//end setFldTypeArray
       public String[] getFldSizeArray(){
         return fieldSizeArray;
      }//end setFldSizeArray 
       public int getNumLines(){
         int countLines = 0;
         try {
                //File must be in same director and be the name of the string below
            BufferedReader in = new BufferedReader(new FileReader("files/FLAT.txt"));
            String str;
            while ((str = in.readLine()) != null) {
               countLines++;
            in.close();
             catch (IOException e) {}    
         return countLines;
      }//end of getNumLines
       public void loadFixedWidthFile(){
         int c = getNumLines();
         int i = 0;
         String[] lineProcessed = new String[c];
         String chars;
         try {
                //File must be in same director and be the name of the string below
            BufferedReader in = new BufferedReader(new FileReader("files/FLAT.txt"));
            String str;
            while ((str = in.readLine()) != null) {
               //System.out.println(str.length());
               lineProcessed[i] = parseThatLine(str);
               i++;
            in.close();
             catch (IOException e) {}     
            //write out the lineProcess[] array to another file
         writeThatFile(lineProcessed);
      }//end loadFixedWidthFile()
       public void writeThatFile(String[] s){
         try {
            BufferedWriter out = new BufferedWriter(new FileWriter("files/outfilename.txt"));
            for(int i = 0; i < s.length -1; i++){
               out.write(s);
}//end for loop
out.close();
catch (IOException e) {}
}//end writeThatFile
public String parseThatLine(String s){
int start = 0;
int end = 0;
String parsedLine = "";
int numChars = getFldSizeArray().length;
//Print number of lines for testing
//System.out.println(numChars);
String[] oArray = getFldSizeArray();
//String chars = oArray[0];
//System.out.println(chars.length());
//oArray
for(int i = 0; i < numChars -1; i++ ){
if(i == 0){
start = 0;
end = end + Integer.parseInt(oArray[i])-1;
else
start = end;
end = end + Integer.parseInt(oArray[i]);
parsedLine = parsedLine + s.substring(start, end) + "~";
}//end for loop
return parsedLine;
}//End of parseThatLine
I have tried to illeminate as many arrays as I can thinking that was chewing up the memory but to no avail.
Any thoughts or ideas?
Message was edited by:
SaipanMan2005

Answer 1

You should not keep a String array of all the lines of the file read.
Instead for each line read, parse it, then write the parsed line in the other file:      public void loadFixedWidthFile() {
         BufferedReader in = null;
         BufferedWriter out = null;
         try {
            //File must be in same director and be the name of the string below
            in = new BufferedReader(new FileReader("files/FLAT.txt"));
            out = new BufferedWriter(new FileWriter("files/outfilename.txt"));
            String str;
            while ((str = in.readLine()) != null) {
               //System.out.println(str.length());
               str = parseThatLine(str);
               //write out the parsed str to another file
               out.write(str);
         catch (IOException e) {
            e.printStackTrace(); // At least print the exception - never swallow an exception
         finally { // Use a finally block to be sure of closing the files even when exception occurs
            try { in.close(); }
            catch (Exception e) {}
            try { out.close(); }
            catch (Exception e) {}
      }//end loadFixedWidthFile()Regards