OutOfMemoryError when retrieving large resultset

Question

In my application I have one type of object (Call it O for now) that has
a lot of data in the database (+50000 rows). Now I need to create an
object of class T that has a m-n binding to 40000 instances of O. In the
database this is mapped to a link table between the table for O and T.
Now I get an OutOfMemoryError when I perform the following code to add
one T to the aforementioned O:
PersistenceManager pm = VRFUtils.getPM(getServlet());
T t = new T();
//Fill arbitrary fields
t.setToelichting(mtForm.getToelichting());
//Add T to a set of O's
Set os = new HashSet();
Query q = pm.newQuery(O.class,"aangemaakt==parmaanmaak");
q.declareParameters("java.util.Date parmaanmaak");
os.addAll(q.execute(field));
t.setOs(os);
//Make T persistent
pm.currentTransaction().begin();
pm.makePersistent(toelichting);
pm.currentTransaction().commit();
pm.close();
After debugging I've found that the OutOfMemoryError occurs even when I
don't make anything persistent, but simply run the query that retrieves
40000 records and do a c.size() on the result.
I'm running Kodo against MySQL using Optimistic transactions.
What I must appreciate, is that the OutOfMemoryError does not upset the
rest of the Kodo system.
Please advise,
Martin van Dijken
PS: The c.size() issue I've been able to resolve using KodoQuery's
setResult("count(this)"), but I can't think of anything like that, that
would apply to this situation.

Answer 1

As you may know, Kodo has several settings that allow you to use large
result sets without running out of memory (assuming your driver supports
advanced features):
http://www.solarmetric.com/Software/Documentation/latest/docs/ref_guide_dbsetup_lrs.html
Kodo also allows you to apply these settings to persistent fields, so
that the entire collection or map field does not reside in memory:
http://www.solarmetric.com/Software/Documentation/latest/docs/ref_guide_pc_scos.html#ref_guide_pc_scos_proxy_lrs
Unfortunately, you are trying to create a relation containing more
objects than your system can handle, apparently, and when creating a
relation you pretty much have to load all the related objects into
memory at once right now.
In the next beta release of 3.1, there is a solution to this problem.
Our next release no longer holds hard references to objects that have
been flushed. So by using the proper large result set settings as noted
above, and by making your relation a large result set relation also as
noted above, and by adding only a few hundred objects at a time from the
query result to the relation and then manually flushing, you should be
able to perform a transaction in which all the queried objects are
transferred from the query result to the large result set relation
without exhausting memory.