XML,CLOB AND MEMORY : CONSUMED BUT NOT RELEASED !!!

Question

Hi,
I'm working with XMLGEN Package and XSLT Processor to produce XML Document on Oracle 8.1.7.3 server, I use 9i XML and Java packages.
I'm facing the following BIG MEMORY problem :
Environment : I must generate an XML parsed with an XSL document of 80Mo (nearly 22 000 rows),
as XMLGEN is on DOM parsing method, I extract my XML by 500 Rows and I loop until 22 000 rows. I use DBMS_JOB with jobs who read and execute export each minute.
The algorithme is :
keeprow=22000
while keeprow>0
Create 3 clob (DBMS_LOB Package) : one for XSL Sheet, one for XML and one for result
GetXML (XMLGEN Package)
Transform in XSL (XSL Processor)
Write to disk (UTL_FILE Package)
Free the 3 Clob ((DBMS_LOB Package))
keeprow =keeprow-500
loop
The problem : The process start at 250Mo ot total memory and END at 1000 Mo of used memory and NEVER RELEASE 1 ko of memory.
So often I get a JavaOutOfMemoryError (I've allocated 1Go Ram to my JAVA process).
Any help will be very very appreciated !
My derived problem is 22 000 rows is not enough I've some export of 200 000 rows to do (And I cannot allocate 10 Go of RAM !!!)
Following My PL/SQL Code.
Regards
Fred
     PROCEDURE DO_EXPORT_XML(
               P_JOB_ID JOB_PARAMETRE.JOB_ID%TYPE,
               P_JOB_ID_ORDRE JOB_PARAMETRE.JOB_ID_ORDRE%TYPE,
               P_CLE_UP UPLOADREQ_TEMP.ID%TYPE,
               P_LOAD_OR_DELOAD VARCHAR2)
          IS
          L_FILE_NAME JOB_PARAMETRE.JOB_FILE_NAME_DEST%TYPE;
          L_REP_NAME JOB_PARAMETRE.JOB_REP_NAME_DEST%TYPE;
          L_FILE_STYLESHEET JOB_PARAMETRE.JOB_STYLESHEET%TYPE;
          L_VERSION_PDM JOB_PARAMETRE.JOB_VPDM%TYPE;
          P_SELECT varchar2(4000):='';
          P_CURSOR varchar2(4000):='';
     l_filehandler_out UTL_FILE.FILE_TYPE;
          --Variable pour le traitement par lot de 500
          L_NBROW_TODO_ATONCE number := 500;
          L_NBROW_MIN number := 1;
          L_NBROW_MAX number := -1;
          L_NBROWKEEPTODO number := -1;
          xslString CLOB := null;
          res number default -1;
          xmlString CLOB := null;
          li_ret number := 0;
          li_faitle number := 0;
          amount integer:= 255;
          li_loop integer := 0;
          charString varchar2(255);
          ls_deload varchar2(255) default '';
          ls_SQL varchar2(4000) default '';
          ls_temp_file varchar2(255) default '';
          text_file_dir varchar2(32) := 'e:\temporarydir';
          l_par xmlparser.parser;
     l_xml xmldom.domdocument;
     l_pro xslprocessor.processor;
     l_xsl xslprocessor.stylesheet;
          docfragnode xmldom.DOMNode;
          docfrag xmldom.DOMDocumentFragment;
          l_parsedclob clob := null;
          l_amount binary_integer := 32767;
          l_ligne varchar2(32767);
          l_offset number default 1;
          l_pos number default null;
          l_pos2 number default null;
          l_lobsize number default null;
          l_memsize number default 1073741824; --1024 Mo
          l_mempipo number default 0;
          type rc is ref cursor;
          l_cursor rc;
          cursor TEMPLATE is select UNSPSC,1 as NB from UPLOADREQ_TEMP where 1=2;
          c1rec TEMPLATE%rowtype;          
          BEGIN
          l_mempipo:=setmaxmemorysize(l_memsize);
          dbms_lob.createtemporary(l_parsedclob, true, dbms_lob.session);
          dbms_lob.createtemporary(xmlstring, true, dbms_lob.session);
          --return the good select
          GET_SELECT_TO_EXPORT_XML(P_JOB_ID , P_JOB_ID_ORDRE , P_CLE_UP , P_SELECT,P_CURSOR, P_LOAD_OR_DELOAD);
                    SELECT JOB_FILE_NAME_DEST,JOB_REP_NAME_DEST,JOB_STYLESHEET
                    INTO L_FILE_NAME,L_REP_NAME,L_FILE_STYLESHEET
                    FROM JOB_PARAMETRE
                    WHERE JOB_ID =P_JOB_ID AND JOB_ID_ORDRE=P_JOB_ID_ORDRE;
                    l_filehandler_out := UTL_FILE.FOPEN(text_file_dir, L_FILE_NAME, 'w',l_amount);
                    --Return XSL Sheet in a clob : cause of memory consumed  but not released
               xslString := METACAT.load_a_file( 1,L_FILE_STYLESHEET);     
                    open l_cursor for P_CURSOR;
LOOP
               fetch l_cursor into c1rec;
               exit when l_cursor%notfound;
                         L_NBROW_MIN := 1;
                         L_NBROW_MAX := 0;
                         L_NBROWKEEPTODO:=c1rec.NB;
                         LOOP
                         begin
                              if(L_NBROWKEEPTODO > L_NBROW_TODO_ATONCE) THEN
                                   begin
                                   L_NBROW_MAX:= L_NBROW_TODO_ATONCE + L_NBROW_MAX;
                                   L_NBROWKEEPTODO:= L_NBROWKEEPTODO - L_NBROW_TODO_ATONCE;
                                   end;
                              else
                                   begin
                                   L_NBROW_MAX:= L_NBROW_MAX + L_NBROWKEEPTODO;
                                   L_NBROWKEEPTODO:=0;
                                   end;
                              end if;
                              --on ouvre le fichier de risultats
                              ls_SQL:= P_SELECT || ' AND ( ROWNUM BETWEEN ' || L_NBROW_MIN || ' AND ' || L_NBROW_MAX || ' ) and UNSPSC=''' || c1rec.UNSPSC || '''';
                              ls_temp_file := c1rec.UNSPSC || '_' || L_FILE_NAME;
                              L_NBROW_MIN:=L_NBROW_TODO_ATONCE + L_NBROW_MIN;
                              --CAT_AUTOLOAD.JOB_ADD_TRACE (P_JOB_ID,'UPLOAD REQUISITE : Export donnies REQUETE ' || to_char(li_loop), ls_SQL,'',0,0);
                              xmlgen.resetOptions;
                              xmlgen.setErrorTag('ERROR_RESULT');
                              xmlgen.setRowIdAttrName('NAH');
                              xmlgen.setRowIdColumn('NAH');
                              xmlgen.setEncodingTag('ISO-8859-1');
                              xmlgen.useNullAttributeIndicator(false);
                              if(xmlString is not null) then
                                   dbms_lob.open(xmlString,dbms_lob.lob_readwrite);
                                   l_lobsize:= dbms_lob.Getlength(xmlString);
                                   if(l_lobsize>0) then
                                   dbms_lob.erase(xmlString,l_lobsize,1);
                                   end if;
                                   dbms_lob.close(xmlString);
                              dbms_lob.freetemporary(xmlString);
                                   dbms_lob.createtemporary(xmlstring, true, dbms_lob.session);
                              end if;
--Return XML in a clob : cause of memory consumed  but not released
                              xmlString := xmlgen.getXML(ls_SQL,0);
                              l_par := xmlparser.newparser;
                              xmlparser.parseclob(l_par, xslString);
                              l_xsl := xslprocessor.newstylesheet(xmlparser.getdocument(l_par),null);
                              xmlparser.parseclob(l_par, xmlString);
                              l_xml := xmlparser.getdocument(l_par);
                              l_pro := xslprocessor.newprocessor;
                                   xslprocessor.showWarnings(l_pro, true);
                                   xslprocessor.setErrorLog(l_pro, text_file_dir || substr(ls_temp_file,0,length(ls_temp_file)-4) || '_logerreur.XML');
                                   if(l_parsedclob is not null) then
                                             dbms_lob.open(l_parsedclob,dbms_lob.lob_readwrite);
                                             l_lobsize:= dbms_lob.Getlength(l_parsedclob);
                                             if(l_lobsize>0) then
                                             dbms_lob.erase(l_parsedclob,l_lobsize,1);
                                             end if;
                                             dbms_lob.close(l_parsedclob);
                                        dbms_lob.freetemporary(l_parsedclob);
                                             dbms_lob.createtemporary(l_parsedclob, true, dbms_lob.session);
                                   end if;
                    --Return XML Processed with XSL in a clob : cause of memory consumed  but not released
                              xslprocessor.processxsl(l_pro,l_xsl,l_xml,l_parsedclob);
                                   --release NOTHING
                              xmlparser.freeparser(l_par);
                              xslprocessor.freeprocessor(l_pro);
                                                  l_ligne:='';
                                                  l_offset :=1;
                                                  l_pos := null;
                                                  l_pos2 := null;
                                                  if(li_loop=0) then
                                                       begin
                                                            --on ouvre le fichier et on sauve l'entete + les donnies dedans.
                                                                 l_pos:=dbms_lob.instr(l_parsedclob,'</DATA>');
                                                  if ( nvl(l_pos,0) > 0 ) then
                                                                      loop
                                                                      if(l_pos-1>l_amount + l_offset ) then
                                                                                l_ligne:=dbms_lob.SUBSTR(l_parsedclob,l_amount,l_offset);
                                                                                UTL_FILE.PUT(l_filehandler_out,l_ligne);
                                                                                UTL_FILE.fflush(l_filehandler_out);
                                                                                l_offset:=l_offset+l_amount;
                                                                           else
                                                                                l_ligne:=dbms_lob.SUBSTR(l_parsedclob,l_pos-1 -l_offset ,l_offset);
                                                                                UTL_FILE.PUT(l_filehandler_out,l_ligne);
                                                                                UTL_FILE.fflush(l_filehandler_out);
                                                                                exit;
                                                                           end if;
                                                                      end loop;
                                                                 else
                                                                      EXIT;
                                                                 end if;
                                                       end;
                                                  else
                                                       --on met les donnies donc on ne repete pas le debut
                                                       begin
                                                                 l_pos:=dbms_lob.instr(l_parsedclob,'<ITEM');
                                                  if ( nvl(l_pos,0) > 0 ) then
                                                                 l_pos2:=dbms_lob.instr(l_parsedclob,'</DATA>');
                                                  if ( nvl(l_pos2,0) > 0 ) then
                                                                      loop
                                                                      if(l_pos + l_amount <= l_pos2 -1 ) then
                                                                                l_ligne:=dbms_lob.SUBSTR(l_parsedclob,l_amount,l_pos);
                                                                                UTL_FILE.PUT(l_filehandler_out,l_ligne);
                                                                                UTL_FILE.fflush(l_filehandler_out);
                                                                                l_pos:=l_pos +l_amount;
                                                                           else
                                                                                l_ligne:=dbms_lob.SUBSTR(l_parsedclob,l_pos2 -1 -l_pos,l_pos);
                                                                                UTL_FILE.PUT(l_filehandler_out,l_ligne);
                                                                                UTL_FILE.fflush(l_filehandler_out);
                                                                                exit;
                                                                           end if;
                                                                      end loop;
                                                                      else
                                                                      exit;                                                                      
                                                                      end if;
                                                                 end if;
                                                       end;
                                                  end if;
                                             li_loop:=li_loop + 1 ;
                                             --UTL_FILE.FCLOSE(l_filehandler_in);
                                             JAVA_GC();
                                             EXIT WHEN L_NBROWKEEPTODO=0;
                                             Exception
                                             when others then
                                                  begin
                                                  -- IF(utl_file.is_open(l_filehandler_in)) THEN
                                                  --               utl_file.fclose( l_filehandler_in);
                                                  -- END IF;
                                                  IF(utl_file.is_open(l_filehandler_out)) THEN
                                                                 utl_file.fclose( l_filehandler_out);
                                                  END IF;
                                                  RAISE_APPLICATION_ERROR(-20001,'File with errors');
                                                  end;
                         END;
                         END LOOP;
END LOOP;
CLOSE l_cursor;
                    if ( xmlString is not null ) then
                              dbms_lob.open(xmlString,dbms_lob.lob_readwrite);
                              l_lobsize:= dbms_lob.Getlength(xmlString);
                              if(l_lobsize>0) then
                              dbms_lob.erase(xmlString,l_lobsize,1);
                              end if;
                              dbms_lob.close(xmlString);
                              dbms_lob.freeTemporary( xmlString);
                    end if;
                    if(l_parsedclob is not null) then
                              dbms_lob.open(l_parsedclob,dbms_lob.lob_readwrite);
                              l_lobsize:= dbms_lob.Getlength(l_parsedclob);
                              if(l_lobsize>0) then
                                   dbms_lob.erase(l_parsedclob,l_lobsize,1);
                              end if;
                              dbms_lob.close(l_parsedclob);
                              dbms_lob.freetemporary(l_parsedclob);
                    end if;
                    UTL_FILE.NEW_LINE(l_filehandler_out);
                    l_ligne:='</DATA></CATALOG>';
                    UTL_FILE.PUT(l_filehandler_out,l_ligne);
                    UTL_FILE.FCLOSE(l_filehandler_out);                    
               EXCEPTION
               when others then
                         begin
                         IF(utl_file.is_open(l_filehandler_out)) THEN
                                   utl_file.fclose( l_filehandler_out);
                              END IF;
                         end;     
          END;
******************************

Answer 1

Thank you for the info - I had no idea I was puing myself in danger by cutting it so close.  Since your post I have moved my iphoto library to an external drive and now have 165 GB of space on my HD.  Following this I have 2 questions.
1.  Since my available HD space was reduced by the size of the photo download it seems logical that the download is somewhere on my HD still.  Is there a place where these photos might be hiding on my HD even though they are not available on the iphoto library?
2.  I was able to recover the .jpg files which are fine.  I also recovered the .mov files but they have been compromised.  I am hoping I can find the originals still on the HD somewhere.  If not, do you have any suggestions for recovery methods or programs?  I have not used the SD card since the incident so I should be able to attempt another recovery to salvage the .mov files if there is an alternative method/program available.
Thanks again!