"heap limit" - or the death of a perfectly sound web application ;-)

Question

hi everyone,
i am running a web app on tomcat 6. There is this specfic feature in the web application where i need to parse data from a csv file and programatically place data in an excel file.
i am using these api's :
jexcelapi
ostermillerutil
here is the code
// Parsing CSV file
          CSVParser p = new CSVParser(in);
          p.setCommentStart("#;!");
          p.setEscapes("nrtf", "\n\r\t\f");
          String[] line = null;
          Label label = null;
          WritableSheet sheet = writeWorkbook.createSheet("Data", 0);     
          int rowCount = 0;
          try {
               while ((line = p.getLine()) != null) {
                    for (int column=0; column < line.length; column++) {
                         try {
                              label = new Label(column,rowCount, line[column]);
                              sheet.addCell(label);
                         } catch (RowsExceededException e) {
                              // TODO Auto-generated catch block
                              e.printStackTrace();
                         } catch (WriteException e) {
                              // TODO Auto-generated catch block
                              e.printStackTrace();                    
                    rowCount++;                    
          catch (IOException e) {
                    e.printStackTrace();
          }the problem is a quite outOfMemory error cause in the while loop.
Its clear that the amout of data (aprox. 26000 rows of data - about 100megs of data) is the problem. My question is any of you can see anything i could optimize the loop with so that the app wont burst ;-)
yours,
Jazz

Answer 1

the problem is a quite outOfMemory error cause in the
while loop.
Its clear that the amout of data (aprox. 26000 rows
of data - about 100megs of data) is the problem. I have a great problem understanding why you need to process 100megs of data in a web application since you are unlikely to be shipping the 100megs to the client for display. I can only see some sense to it if your web application is offering a conversion service but even then you will still have an upload of 100megs followed by a download of 100megs. If you are offering a conversion service then because of the memory requirement you will probably be limited to just one client at a time anyway.
Not what you would call a sound scalable solution to a problem.