When are return values' memory released?

Question

I have been trying to optimize a particular query and have run out of ideas. In this query I have user defined functions which returns about 1600 results per document. I have tried running it with live values and lazy results from both my Java app and the dbxml shell both have similar behavior. If I run 1 document it runs in about 8 seconds and memory continuously climbs to a high of about 350mb which occurs at the end of the query run. When I call delete on the XmlResult object all of the memory is released.
Now if I do the query over two documents the memory climbs to about 400mb and levels off there at which point it seems the results come back significantly slower as if the database is doing something else; maybe determining where it can free memory. Overall it takes the query about 90 seconds to complete. If I try something like:
local:run-query( (: will look in doc1 :) .... ) | local:run-query( (: will look in doc2 :) .... ) instead of local:run-query( (: will look in doc1 and doc2 :) .... )
The performance is more of what I would have expected: runs in 17 seconds and continuously climbs in memory to about 700mb which occurs at the end of the query.
So I guess my question would be is this behavior by design and why does it hang on to so much memory even though I do delete on the XmlValue result as soon as I get it back?
I can post the query and give an idea of how the data is formatted if it would help. Any insight that anyone has will be helpful.
I should note that I am using version 2.3.10 on windows xp. The machine is a Pentium 4 3.00GHz with 1GB of ram
Thanks in advance,
Pralit
Message was edited by:
user580792

Answer 1

George,
My documents are about 11MB as an xml file (should be about 180,000 elements). For the
moment there are only 2 documents in my container although it could get as high as 30 but not
typically. With 2 documents the size of the container is 56MB. I create the environment in Java
using the following config:
EnvironmentConfig envConfig = new EnvironmentConfig();
envConfig.setAllowCreate(true);
envConfig.setCacheSize(100 * 1024 * 1024 );
envConfig.setInitializeCache(true);
envConfig.setInitializeLocking(true);
I have tried a cache size of 500mb but didn't notice much of a change in performance.
I feel you are right about my query defeating the lazy evaluation. All the looping for the sums in
the get-weighted-eff function seems to be the cause for a lot of the memory use. Also the newly
created element results would probably not be lazily returned (although these seem to get
deleted promptly).
Anyways my query and an example of the data follow . Any suggestion on what I can do to
reduce the memory if at all possible would be great. As I said before I am out of ideas.
Thanks,
Pralit
The Data:
<scenario name="Keyword-All" date="2007-8-11T18:46:52-00:00'">
<world>
  <region name="USA" type="region">  <!-- 14 per document, Global implies all regions -->
   <supplysector name="electricity" type="sector">  <!-- 32 sectors per region -->
    <subsector name="nuclear" type="subsector">  <!-- 1-4 per sector -->
     <technology year="1990" name="Gen_II_LWR" type="technology">  <!-- 3-7 per subsector -->
      <output-primary name="electricity" type="output">  <!-- could be up to 3 sets of inputs/ouputs per technology -->
       <physical-output unit="EJ" vintage="1990">2.20172</physical-output>
      </output-primary>
      <input-non-energy name="non-energy" type="input">
       <price-paid unit="1975$/GJ" vintage="1990">5.8</price-paid>
       <IO-coefficient unit="unitless" vintage="1990">1</IO-coefficient>
      </input-non-energy>
      <input-energy name="nuclearFuelGenII" type="input">
       <price-paid unit="1975$/GJ" vintage="1990">0.20381</price-paid>
       <demand-physical unit="EJ" vintage="1990">6.61177</demand-physical>
       <IO-coefficient unit="unitless" vintage="1990">3.003</IO-coefficient>
      </input-energy>
      <keyword primary-renewable="nuclear-elect"/>  <!-- keywords are only in select technologies about 2000 total in a document -->
     </technology>
....The Query:
declare function local:get-weighted-eff($outputs as node()*) as node()* {
   unordered {
     for $year in distinct-values($outputs/@vintage)
     let $eff_unweight := sum(for $res in $outputs[@vintage = $year]
                       let $eff_curr := $res/parent::*/following-sibling::input-energy/IO-coefficient[@vintage = $year]                       return $res div $eff_curr),
             $weight := sum($outputs[@vintage = $year]),
             $eff_weighted := $eff_unweight div $weight
     return element efficiency-weighted {attribute year { $year }, text { $eff_weighted }}
declare function local:append-heirarchy($parent as node(), $append as node()) as node() {
let $scn := $parent/ancestor::scenario,
          $rgn := $parent/ancestor::region
   return 
          document { element scenario { $scn/@*,
                                                element region {
                                                        $rgn/@*,
                                                        $append
(: I can get by with just the scenario and region
let $new_node := element {local-name($parent)} {$parent/@*, $append}  
return 
if(local-name($parent) != 'scenario')
then local:append-heirarchy($parent/parent::*, $new_node)
else document { $new_node } :)
declare function local:get-primary-renewable($outputs as node()*, $weighted_effs as node()*) as node()* {
unordered {   
for $output in $outputs
let $new_output :=   element input {
         attribute type {'input'},
         attribute name {$output/parent::*/following-sibling::keyword/@primary-renewable},
         element demand-physical {
                 attribute vintage {$output/@vintage},
                 attribute unit {$output/@unit},
                 text { $output div $weighted_effs[@year=$output/@vintage] }
$new_root := local:append-heirarchy($output/parent::*/parent::*, $new_output)
return $new_root//text()
declare function local:run-primary-energy($scenarios as xs:string*, $regions as xs:string*, $collection as xs:string) as node()* {
unordered {   
let $regionsG := if(not($regions[1] = 'Global'))
                  then $regions                  else distinct-values(collection($collection)/scenario/world/region/@name)
return 
for $scenario in $scenarios,      
$region in $regionsG  
let $scenario_split := tokenize($scenario, ' '),                          
$currTree := collection($collection)/scenario[@name = $scenario_split[1] and @date = $scenario_split[2]]/world/region[@name=$region]
return (: get renewables from electricity :)
local:get-primary-renewable($currTree/supplysector[@name='electricity']//keyword[fn:exists(@primary-renewable)]/preceding-sibling::output-primary/physical-output,
   local:get-weighted-eff($currTree/supplysector[@name='electricity']//keyword[fn:exists(@average-fossil-efficiency)]/preceding-sibling::output-primary/physical-output))        
| (: get renewables from H2ProdCS :)
local:get-primary-renewable($currTree/supplysector[@name='H2ProdCS']//keyword[fn:exists(@primary-renewable)]/preceding-sibling::output-primary/physical-output,
   local:get-weighted-eff($currTree/supplysector[@name='H2ProdCS']//keyword[fn:exists(@average-fossil-efficiency)]/preceding-sibling::output-primary/physical-output))        
| (: get renewables from H2ProdDist :)
local:get-primary-renewable($currTree/supplysector[@name='H2ProdDist']//keyword[fn:exists(@primary-renewable)]/preceding-sibling::output-primary/physical-output,
   local:get-weighted-eff($currTree/supplysector[@name='H2ProdDist']//keyword[fn:exists(@average-fossil-efficiency)]/preceding-sibling::output-primary/physical-output))        
| (: get the primaries :)
$currTree//keyword[fn:exists(@primary-consumption)]/preceding-sibling::input-energy/demand-physical/node()
local:run-primary-energy(('Keyword-All 2007-8-11T18:46:52-00:00'), ('Global'), 'temp2.dbxml')