More jrun using up memory problems

Question

There seem to be a lot of problems like this but everyone's situation seems to be a little different.  It makes it hard to find solutions.  Here is what we are dealing with.
We are taking an XML file containing product data and importing it into a SQL 2000 database.  The xml file is about 6mb and it has product data for about 4700 products.  When we wrote the import we were running coldfusion 7 standard updated to 7.02.  We run the import every night and it seemed to run fine.  It took about 45 minutes to run but the server didn't crash.  We just recently upgraded to cf 9 standard.  After the upgrade when the import ran we were seeing lots of out of memory errors in the log.  The second night it ran coldfusion had to be restarted.
We setup a testing server and ran the import there and watched the memory usage of jrun.  As the import ran memory usage kept climbing until it reached the max heap size.  After this the import got very slow and eventually coldfusion became completely unresponsive.
We changed the import so that it would do the products in smaller batches as separate requests.  This seemed to be the answer.  Between the requests the memory usage would drop down some and the whole thing was able to run without using up all the available memory.  The import also takes a lot less time now that the memory usage isn't so high.  We thought we had it fixed so we started running it on the production server.
After a couple days we noticed that our memory usage was very high on the production server.  We went back to the testing server and ran the import a few times.  After this we could see that each time it was run the memory usage would increase.  After about 4 times the memory reached the max heap size and coldfusion became unresponsive.  The memory usage never goes back down.

Answer 1

Some things to check / consider.
1. What are your JVM settings?  What is your Maximum JVM Heap Size?  If you are using the default value, 512 MB, consider a larger value.  On a 32 bit server the upper limit is 1.8 GB.  A 64 bit server can allocate more memory to the JVM, but I haven't seen any documentation that specifies the upper limits on 64 bit.  (If anyone can point me to some info on this I'd be grateful.)
2. Consider moving the XML to SQL import process off of ColdFusion to another tool.
Some options:
a. A SQL Server DTS package
b. Use the bulk insert feature of SQL Server
c. Use a third party tool such as Altova Mapforce
Java and JVM page in CF administration
http://help.adobe.com/en_US/ColdFusion/9.0/Admin/WSc3ff6d0ea77859461172e0811cbf3638e6-7ffc .html#WSc3ff6d0ea77859461172e0811cbf3638e6-7feb
"Maximum JVM heap size greater than 1.8GB will prevent ColdFusion MX from starting"
http://go.adobe.com/kb/ts_tn_19359_en-us
"Examples of Bulk Importing and Exporting XML Documents" (SQL Server 2000)
http://msdn.microsoft.com/en-us/library/ms191184.aspx
Altova Mapforce
http://www.altova.com/mapforce.html
"Troubleshooting a Leaky Heap in Your JVM" (blog entry)
http://www.coldfusionmuse.com/index.cfm/2008/2/12/leaky.heap.jvm