XML parsing huge file

Question

Hi,
I have a 36M XML file i need to parse, I'm new to XML.
I usually get a 200K file in CSV format from most of my client that they transfer into there account i then simply update the MSSQL database with the CSV file at midnight on my server. But now i have 74 clients that are regroup and they send me 1 XML file.
When i run it using the sample they gave me it works fine but on the 36M file i get a Jrun error then i found out that :
<CFFile action="READ" variable="xmlfile" file="c:\mypath\#clientfile#.xml" charset="utf-8">
<cfset xmlObj = xmlParse(#xmlfile#)>
Doesnt work on big files because it runs out of memory.
I need a way to parse that file using Java i downloaded xmlsax.js but i dont know how to use it to parse then get my parsed var back from it can anyone help me please.
I got the file here :  http://xmljs.sourceforge.net/website/sampleApplications-sax.html
Thank you

Answer 1

In response to Owain Norths' comments about DOM parsing.
I'm not sure if the memory issues are the fault of the DOM parsing method being used or if the problem is in how CF converts XML text into CF objects (arrays, structs) that the XML text represents.  It possible that the CF objects are responsible for using excessive amounts of memory.  Either way it sounds like CF's XML parsing capabilities aren't appropriate for larger (large being a relative term) XML files.
It might be an interesting experiment to use third party Java components (such as Xerces2) to parse some XML files and see what the performance and memory usage look like.
I will re-state my original advice.  The poster needs to import data from XML files into tables on MS SQL Server.  Bulk import tasks, such as from XML or CSV files, are generally better handled in MS SQL Server.  Some options include: a job that executes T-SQL, an Integration Services package, or the Bulk Copy Program (BCP) utility. 
From: Owain North [email protected]
Sent: Fri 11/12/2010 8:57 AM
To:
Subject: XML parsing huge file
Couldn't agree more, and to be honest I can't believe this hasn't come up before. To me, the thought that something like CF should have to be bypassed when you get to files of a few megs is utterly ridiculous. I haven't looked into the different methods of parsing XML as it's really not my thing, but are we saying that DOM parsing is necessary for CF to be able to perform the functions it does on  the resulting XML object? Or does one create the same result, just through a different method?
Owain North
Code Monkey
Titan Internet Ltd
http://www.titaninternet.co.uk <http://www.titaninternet.co.uk/>
Owain North is a mildly overweight computer programmer who likes to sit in the corner of a darkened room tapping away on his keyboard whilst wearing a massive set of headphones to avoid human contact where possible. He particularly likes to avoid natural light and salad.
In his spare time he likes to pet his dog and work on his track car: http://www.306gti6.com/forum/showthread.php?id=124722&page=1
The other day he went up to the toilets upstairs and there were no hand towels left! Bad times.
It's Filthy Friday, so we all got Dominos for lunch. Large (obviously) half and half Mighty Meaty and American Hot. Good it was, especially as one of the other guys didn't want his garlic & herb dip = win.
At the moment, he's having to look into WCF for a new project on  server monitoring. He doesn't know anything about it yet but after a  quick session on Amazon with the company credit card and some  extortionate delivery fees he's well on his way to writing his first WCF  service.
In case you're interested - in the end, he just had to dry his hands on his jeans.