Parsing an XML file to a table

Question

Hi,
I am looking at a scenario where right now we have a daily data feed that is an xml file that is approx between 1 to 2 GB in size. Using Java code, this file is parsed and one by one, the records are inserted into the DB (Oracle 10.2). But as the data feed is growing larger, so is the file size -- which is resulting in more time taken for this process to run.
I do realize that the ideal scenario would be to stop using the file as a source of feed. But this is beyond our control.
I looked at using sqlLoader but as I understand it requires a csv based file. In my case, I would have to convert the xml file to csv, which again might be a huge operation to be carried out daily. (Edit: Am I wrong? can we use Sql Loader here?)
Thus I am exploring options if there could be a way to write a PL/SQL procedure that would parse after taking this xml file as an input and load the data into a table. Any suggestions/directions will be hugely appreciated.
Thanks,
Ak
Edit:
I also browsed through a few questions asked previously here and I came across this: Import Large XML File to Table
My situation is similar and as per the suggestion here by A_Non, SECUREFILE BINARY XML storage can be used. But the problem I have is that I am on Oracle 10.2 while this option is only available in a later version, if I am right.
Edited by: Aditya Kumar on Nov 21, 2012 10:49 PM
Edited by: Aditya Kumar on Nov 22, 2012 12:20 AM

Answer 1

Thanks a lot for your reply Odie.
Exact Oracle version is Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi
A couple of doubts here:
1. The file size would range 1-2 GB (and can go beyond) Is SQLLoader also an option feasible in this scenario?
2. The link you mentioned has this line: Storing the XML document in Oracle Database using Binary File (BFILE) -- How is this different from Binary XML in 11g? I am confused.
3. With respect to storage space usage, should we be concerned about the storage space/ memory that the operation of loading the file in this (temporary) table will take? Because right now, the xml feed file is residing in the webserver where java code iterates through it and inserts each record in the table. Now this will change and technically this file will have to go in the DB box.
We do not have the xsd for it as right now we retrieve a few selected tags from this xml, parse and store it in a table (columns mapped to tags). Below is the sample xml. This is one such record and there are thousands below it. I had to change things because of data confidentiality but it mimics the original.
<?xml version="1.0" encoding="windows-1250"?>
<Involvements xmlns="com/xyz/us/abc/v0/ijkdatatypes" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ijk:Involvement xmlns:ijk="com/xyz/us/abc/v0/ijkdatatypes">
<ijk:PCenterData>
<ijk:HPCenterData>
<ijk:HPCenter/>
<ijk:HPCName/>
</ijk:HPCenterData>
</ijk:PCenterData>
<ijk:BillableWBSCode>00000000</ijk:BillableWBSCode>
<ijk:EDescription/>
<ijk:InvolvementType>0</ijk:InvolvementType>
<ijk:ProductData>
<ijk:ICode/>
<ijk:Product/>
<ijk:ProductName/>
</ijk:ProductData>
<ijk:PartnerData>
<ijk:BPartnerData>
<ijk:BPartnerNumber/>
</ijk:BPartnerData>
<ijk:PPartnerData>
<ijk:PPartnerNumber/>
</ijk:PPartnerData>
<ijk:LeadPartnerData>
<ijk:LeadCSPNumber/>
</ijk:LeadPartnerData>
</ijk:PartnerData>
<ijk:ManagerData>
<ijk:BManagerInformation>
<ijk:BManagerNumber/>
</ijk:BManagerInformation>
<ijk:PerformanceManagerInformation>
<ijk:PerformanceManagerNumber/>
</ijk:PerformanceManagerInformation>
</ijk:ManagerData>
<ijk:InvolvementDateData>
<ijk:InvolvementstartDate/>
<ijk:InvolvementTerminationDate/>
</ijk:InvolvementDateData>
<ijk:HostRegion/>
<ijk:WBSD>
<ijk:WBSC>00000000</ijk:WBSC>
<ijk:WBSDescription/>
<ijk:ContractCode/>
<ijk:ContractLine/>
<ijk:ContractLineDescription/>
</ijk:WBSD>
<ijk:PDIndicator>false</ijk:PDIndicator>
<ijk:CompanyCode/>
<ijk:ClientCode/>
<ijk:ContractCode/>
</ijk:Involvement>
</Involvements>