CFC method memory problem

Question

Hi all,
I use CFCs a lot, so when I was creating a tool to migrate
some database tables, I built the actual data transfer function as
a method of a CFC. The basic code just reads a block of 1000
records from the old db, does some minor manipulation of the data,
and then inserts the records into the new db a record at a time
inside a cfloop. Lather, rinse, repeat, until all of the 500K+
records have been transferred.
I was surprised to find that this leads to Java out of memory
errors around the 200Kth record (given my current JVM memory
settings). Having run across a similar problem when sending
broadcast emails from a CFC method, I simply moved the code out of
the CFC method and into the calling cfm page, and it ran through
all of the records without any problems.
The only explanation I can think of for this behavior is that
when the code is inside a CFC method, the query object used to read
the blocks of 1000 records keeps getting reallocated rather than
reused, so when it gets to the point of trying to store 200 blocks
of 1000 records in memory, it runs out of room. If that's the case,
it would seem to be a significant bug that needs to be fixed.
Does anyone know if that's what is really going on?
Thanks,
Russ

Answer 1

> Actually, there is a good reason to put the code in a
cfc: there are almost 100
> tables to migrate, so the code gets rather long and
cumbersome to wade through
> if it isn't broken up.
Fair enough. Youd didn't really tell us that part before,
though.
So you have a calling template which calls an equivalent to
this function
for 100-odd tables? Yikes. I'll reiterate my question whether
this can't
possibly be done DB to DB or via a bulk insert.
> thing, but ultimately my question really has nothing to
do with this particular
> tool. I would really just like to understand why the out
of memory issue exists
> when the code is in a CFC method versus when it is in a
CFM. This issue has
Well it kinda is about this particular tool, because it's
this code that's
causing the problems (or something about the calling code, or
something
like that). Obviously there's some idiosyncasy of it which
gives you
problems when it's in a CFC method as opposed to mainline
code, but I would
not immediately say it's a generic problem with CFCs vs CFMs.
Just yet.
Is this the first table that's being migrated? Or is it
buried somewhere
within the other 100-odd? Is it always this one, or can it
happen on any
of them?
Can you inspect the memory usage between each table's
function, and see if
there's anything unexpected going on (other than it running
out of memory,
which is - in itself - unexpected!).
Do you have 100-odd methods (one for each table) in the CFC,
or lots of
individual CFCs?
I guess you could call a GC in between each table's
processing, and see if
you can keep check on memory usage that way. Or maybe check
memory levels
within that loop you've got, and if it starts flaring up, doa
GC then.
It's not advised to call GCs "by hand", but I have found it's
helped some
times.
Have you sued CF's server monitor or something like
FusionReactor to check
if anything unexpected is consuming RAM?
I think you should try adding <cfqueryparam> tags to
your queries: CF might
be cachinng some info about them, and that could be leaking
memory or
something. Maybe try <cfobjectcache action = "clear">
between each method
callor something.
I wonder if there's something about the fact you're replacing
oldInfo with
a new query within a loop that has a condition based on
oldInfo's record
count. Possibly this is forcing CF to maintain the previous
oldInfo
queries in the background, so you're not actually overwriting
the previous
one with the next one when you requery, you're actually just
getting a new
pointer assigned. Do me a favour, and do this:
<cfset iRecCount = OldInfo.RecordCount>
<cfloop condition="iRecCount GT 0">
<!--- Load the old info --->
<cfquery name="OldInfo" datasource="dsOld"
maxrows="#GetRecsCount#">
</cfquery>
<cfset iRecCount = OldInfo.RecordCount>
</cfloop>
This is a slightly far-fetched notion, but it doesn't hurt to
try: it's a
pretty easy change.
> come up enough times now that I'd like to determine if
there is a workaround.
> Due to the deadline I'm on, I haven't tried simply
assigning the query object
> to an empty string at the end of each iteration yet, but
I will try to do so
> soon so I can report on the result.
It might pay to apply a test rig to this notion first:
0) on a dev machine without any other activity
1) grab the heap usage
2) load a big query in
3) grab the heap usage (should losely reflect the fact you've
just stuck a
chunk of data in RAM)
4) set the query variable to be an empty string
5) grab the heap usage. Check to see if the memory is
reclaimed straight
away.
You might need to do a GC between 4+5 for it to take effect.
I speculate
that the RAM won't get freed up straight away, and you might
be consuing it
faster than the JVM can clear it.
Adam