Best way to serialize large document to XML file

I'm kind of new to XML programming, and am having trouble with streaming very large
content to an XML file. Basically, I need to read/validate/convert a large CSV
file to an XML document. I've had success for small documents using JAXP, but
encountered problems when the size of the files got larger.
Instead of loading the entire CSV file in memory, validating and outputting the
entire document to a XML file, I want the ability to read a single line, validate
it, convert it to an element and output the single element to XML. The problem
is that with Xerces, DOM, etc. serialization routines, they don't give me the
ability to control the writing of elements in the desired way.
For example,
<Parent>
<Child> (represents 1st record of CSV)
</Child>
<Child> (represents 2nd record of CSV)
</Child>
</Parent>
I want the ability to
1) stream the Parent start tag to the XML file
2) create an element from the content in the CSV record
3) stream the new element to the XML file
4) repeat steps 2-3 until end of CSV file
5) write the Parent end tag
It seems like all serializers don't allow this behavior. They only allow a complete
element to be written. Since I don't have a complete Parent element in memory,
this will not work for me. Will stax work for this particular problem?

Hi Joe,
What kinds of problems were you getting with JAXP when you scaled up the
size of the doc?
StAX, it seems, would not help in the generation of the XML, but could
you not create a DOM for each child and write them separately, enclosing
the entire doc in the parent tags? Just a suggestion.
Thanks,
Bruce
Joe Miller wrote:
>
I'm kind of new to XML programming, and am having trouble with streaming very large
content to an XML file. Basically, I need to read/validate/convert a large CSV
file to an XML document. I've had success for small documents using JAXP, but
encountered problems when the size of the files got larger.
Instead of loading the entire CSV file in memory, validating and outputting the
entire document to a XML file, I want the ability to read a single line, validate
it, convert it to an element and output the single element to XML. The problem
is that with Xerces, DOM, etc. serialization routines, they don't give me the
ability to control the writing of elements in the desired way.
For example,
<Parent>
<Child> (represents 1st record of CSV)
</Child>
<Child> (represents 2nd record of CSV)
</Child>
</Parent>
I want the ability to
1) stream the Parent start tag to the XML file
2) create an element from the content in the CSV record
3) stream the new element to the XML file
4) repeat steps 2-3 until end of CSV file
5) write the Parent end tag
It seems like all serializers don't allow this behavior. They only allow a complete
element to be written. Since I don't have a complete Parent element in memory,
this will not work for me. Will stax work for this particular problem?

Similar Messages

What the best way for STORAGE the documents from DMS?

Please, can someone tell me if the limit of storage the originals through a data carrier is two for document or i can to large this limit? 'Cause i know that using Kpro i don't have limit of storage, but i'm trying to use the data carriers 'cause i just have a place in the company server for storage this originals.
I'm trying to send the originals from a path like this
abcs01\dms$\ through of a configuration "servidor, front ends", but i'm not get to send the originals from CV01N for this place, even the document been save. I verify that the documents were not sent when i enter from the CV02N in the same document and try to change or screen the originals, 'cause i receive a error message saying that the SAP doesn't get to find the originals on the path indicated.
Did someone just work on a likely situation?
Did someone have a view about the best way to storage this documents in this case?
If the best way will be KPRO, Do I use KPRO with the Content Server or KPRO with storage the documents in a SAP table through of the standard system aplication "DMS_C1"?
Any idea will be welcome and reward by good points!
Best Regards.

Hi,
The decision to go for either content server or vault depends on what data will be stored. If you are storing drawing data it is highly recommended to go for content server.
For vault storage performance issues are reported.
DMS_C1 is IDES based BSP database repository. It is useful only for dev/testing.
You will be required to create storage repository for CS.
If you have server available when you insert content server CD inside it, it automatically asks you for maxDB database instance creation. As you click on next it will take you further to create maxDB instance. K-Pro is service on web server. It is used to manage the checkin-checkout and other functionalities of the CS server data.
BR,
Anirudh,
reward if useful.

What is the best way to create business documents in CRM

Hi All,
What is the best way to create business documents like contract, sales order, debit memo etc in CRM ? Unlike R3 we can't use our good old BDC with recording. Moreover for most of them although there are Business Object but no BAPI to creation so what is the way ? I found in SDN there are two MAGIC Function module CRMXIF_ORDER_SAVE. Do I need to that alawys ?
Is it nees to via IDoc and cannot be done just by calling from ABAP program ? The input parameter of the FM is a complex deep structure.
Please help.

Ashim,
Try looking at the program:
CRM_TEST_ORDER_MAINTAIN
I think that should help you figure out the parameters.
Good luck,
Stephen

What is the Best way to move large mailboxes between datacenters?

What is the Best way to move large mailboxes between datacenters?

Hi,
Are you asking with regards to on-premises Exchange? With Microsoft Online SaaS services (aka Exchange Online) there is no control and no need to control which data center a mailbox resides in.
With regard to on-premises Exchange, you have two choices: you can move it over the WAN in which case you would either do a native mailbox move (assuming you have Exchange 2010 or later you can suspend the move after the copy so you can control the
time of the cutover) or create a database copy in the second data center and once the database copies have synchronized change the active copy.
The other choice is to move is out of band which would usually involve an offline seed of the database (you could conceivably move via PST file but that would disrupt access to the mailbox and is not really the 'best way').
In general, Exchange on-premises questions are best asked on the Exchange forum: http://social.technet.microsoft.com/Forums/office/en-US/home?category=exchangeserver
Thanks,
Guy

What is best way dealing with large tiff file in OSX Lion?

I'm working with large tiff file (engineering drawing), but preview couldnt handle (unresponsive) this situation.
What is best way dealing with large tiff file in OSX Lion? (Viewing only or simple editing)
Thx,
54n9471

Use an iPad and this app http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=400600005&mt=8

Best way to copy large folders to external drive?

What is the best way to transfer large folders across to external drives?
I'm trying to clean up my internal hard drive and want to copy a folder (including its sub folders) of approx 10GB of video files to an external firewire drive.
I've tried drag and drop in finder (both copy and move) and it gets stuck on a particular file and then aborts the whole process.
1. Surely it should copy everything it can, and simply not copy the problem file.
2. Is there a way I can do copy and verify, a bit like when I burn a disk so I can be sure the video files have transferred safely before I delete them from my internal drive?
Many thanks in advance for any advice.

What you are trying to do makes perfect sense to me and I have done the same prior to getting myself a Time Machine system in place.
1. Surely it should copy everything it can, and simply not copy the problem file.
The fact that it is getting stuck on a particular file suggests that there is a problem with it. Try to identify which one it is and deal with that file on it's own. It could be that there is a disk error where that file is stored.
2. Is there a way I can do copy and verify....
The copy process you are using does that implicitly as I understand it.
Chris

What is the best way to import word documents from a PC to the Ipad2

What is the best way to import word documents from a PC to the Ipad 2?

You first need an app on the iPad that supports word documents - the iPad doesn't have a file structure like a 'normal' computer, and every file/document needs to be associated with an app. If you don't have such an app then options include Apple's Pages app and third-party apps such as Documents To Go and QuickOffice HD. How you then get the documents on the iPad will depend upon the app that you have/get as different apps may use different methods e.g. via the file sharing section at the bottom of the device's apps tab when connected to iTunes, via wifi, email, dropbox etc.

HT3847 How is the best way to separate copied MP3 from AIFF files in my library?

How is the best way to separate duplicate MP3 from AIFF files in my library?

Zice wrote:
I want higher resolution then afforded in the original download.
Then why are you converting iTunes purchases?
You cannot get higher resolution by converting the original. This goes for converting anything, not just iTunes purchases.
Creating an AIFF will simply make the file 10 time as large with zero increase in quality.
Don't really want to debate value of creating the new version.
Agreed.
You are simply wasting time and drive space converting iTunes purchases to AIFF.

Best way to spool DYNAMIC SQL query to file from PL/SQL

Best way to spool DYNAMIC SQL query to file from PL/SQL [Package], not SqlPlus
I'm looking for suggestions on how to create an output file (fixed width and comma delimited) from a SELECT that is dynamically built. Basically, I've got some tables that are used to define the SELECT and to describe the output format. For instance, one table has the SELECT while another is used to defined the column "formats" (e.g., Column Order, Justification, FormatMask, Default value, min length, ...). The user has an app that they can use to customize the output...which leaving the gathering of the data untouched. I'm trying to keep this formatting and/or default logic out of the actual query. This lead me into a problem.
Example query :
SELECT CONTRACT_ID,PV_ID,START_DATE
FROM CONTRACT
WHERE CONTRACT_ID = <<value>>Customization Table:
CONTRACT_ID : 2,Numeric,Right
PV_ID : 1,Numeric,Mask(0000)
START_DATE : 3,Date,Mask(mm/dd/yyyy)The first value is the kicker (ColumnOrder) as well as the fact that the number of columns is dynamic. Technically, if I could use SqlPlus...then I could just use SPOOL. However, I'm not.
So basically, I'm trying to build a generic routine that can take a SQL string execute the SELECT and map the output using data from another table to a file.
Any suggestions?
Thanks,
Jason

You could build the select statement within PL/SQL and open it using a cursor variable. You could write it to a file using the package 'UTL_FILE'. If you want to display the output using SQL*Plus, you could have an out parameter as a ref cursor.

Best way to import a 200GB single dump file

I was given a 200GB size single dump file containing full export of a schema. can any please tell me whats the best way to import such a huge dmp file. I need to get this import done asap in QA for testing which will let us solve some production issues. step by step instructions if possible would be really helpful for me to complete my task asap.
Thanks in Advance,
krim.

Hi Krim,
Our dump files normally are never that big so that maybe you could face some other issue here.
If your dump was a full DB schema dump like:
$ exp username/password file=full.dmp parameter-list
then the import should first drop the user in the target system
SQL> drop user username cascade;
this is to drop the existing schema before importing
SQL> Create user according to your reqs
$ imp username/password file=full.dmp full=y commit=y ignore=y
Don't know which env you have to run this, but in our case for instance using an 8 X 1.6GHz Itanium2 Montecito a 14 GB dump takes about a couple of hours to import (with an EMC Clariion disk array). It's also true that Oracle imp (did you use exp or expdp ?) is not able as far as I understand to achieve parallelism like impdp where in case of multiple huge tables the import time could be sped up.
Another thing you may want to check is if you have archive logging on, since the import will log there consuming time.
Cheers,
Mike

What is the best way to handle duplicate in an XML document?

I have an XML document that may contain duplicated nodes. I want to insert it to the DBXML database, so that the duplicated nodes are eliminated.
What would be the best way (in term of performance) to do it?
I thought of enforcing the uniqueness constraint and then insert the nodes one by one, so that I will get an exception from the database if the node is duplicated, but I may have more than 50000 nodes in the worst case, so I'm not sure if this is still a good way to do it.
Can someone give me some suggestion on this?
Thanks!

Hi,
I would suggest to reconsider building of your document so that it doesn't contain duplicates if you don't need them. And it doesn't have much to do with DB XML then.
Also you could insert your document with duplicates and use XQuery Update facilities to delete undesirable nodes.
Vyacheslav

Best Way to Copy Large Files

In copying files to a USB memory stick, is there a particular file size limit above which simple drag and drop using the Finder does not work consistently? In general can we assume that if it appears to copy OK, then the files are in good shape? For example, if I'm copying a Documents folder with 5 GB of data onto an 8 GB USB stick would it be appropriate to use drag and drop? What are the alternative ways to copy large blocks of data? Is there a way to use Disk Utility for this?
I should mention one of the reasons I'm asking. I've occasionally noticed large copying procedures sometimes cause the Finder to "hang."
I do have SuperDuper which I use to clone my internal HD to a peripheral HD on a regular basis. Would that software provide a way to copy single files or folders containing lots of files to my USB memory stick?
Thanks,
Steve M.
Message was edited by: Steve M.

Drag and drop is fine. There's really no good reason to do it a different way.
The only file limit is if the USB stick is formatted FAT32. Then 4GB minus 1 byte is the largest file size it can hold. If you format the stick as Mac OS Extended (HFS+), there's no limit to the file size, just the capacity of the drive itself.
If you need to share between Mac and Windows, then formatting the stick as ExFAT is best. The only catch is that versions of OS X earlier than Snow Leopard will not be able to read the drive. Windows XP, Vista and Win 7 can read ExFAT. There is also essentially no file size limit with ExFAT.
As far as the hanging, that may be the OS stalling on a file larger than 4 GB if the USB stick is formatted as FAT32.

Best way to fill CLOB with external XML from URL

There seem to be a number of great tools available for pl/sql developers in the 8i DB environment and I am tryng to get a handle on them. So far I have worked with XSQL servlet (not PL/SQL), XML Parser and XSU.
The problem I have is to fill a clob from an external xml file via a url and then make that clob available for DBMS_XMLSave to insert into DB. What is the best way to do this? I am thinking maybe I have to use utl_http or utl_tcp to fill the CLOB since I can't seem to find a method in the pl/sql XDK?
XSU - can easily generate CLOB from a query but in this case it is not a query, it is XML from url
XMlparser - can read in from a url using parse() but the document returned is a DOMDocument rather than CLOB.
-quinn

There seem to be a number of great tools available for pl/sql developers in the 8i DB environment and I am tryng to get a handle on them. So far I have worked with XSQL servlet (not PL/SQL), XML Parser and XSU.
The problem I have is to fill a clob from an external xml file via a url and then make that clob available for DBMS_XMLSave to insert into DB. What is the best way to do this? I am thinking maybe I have to use utl_http or utl_tcp to fill the CLOB since I can't seem to find a method in the pl/sql XDK?
XSU - can easily generate CLOB from a query but in this case it is not a query, it is XML from url
XMlparser - can read in from a url using parse() but the document returned is a DOMDocument rather than CLOB.
-quinn

What is the best way to send large, HD video files from the iPad?

Hello,
My department is looking to purchase between 50 and 150 new iPads. We hold conferences where participants take short, 3-minute videos of themselves in situations they have been trained for. The idea was that after they have taken the video, they could simply email these videos to themselves, negating the need to hook the device up to a computer, transfer the video over to a flash drive, and ensure that drive gets back to the right person.
Now that I have the first one, I'm finding any video longer than 45s or so can't be sent via email because it's too large.
Are you kidding?
The file I was just able to send was 7.5MB. That's the biggest video I can send from this thing? Why even give us the ability to take HD video if we can't *do* anything with it?
But, I digress.
What I'm looking for is the best way to get video off the iPads and into the hands of our participants. Are there apps on the App Store which might convert whatever video we take to an emailable size? I thought about setting everyone up with Apple IDs and having them use iMessage since there is no limit on file size, but of course you must own an Apple device in order to do that, and it still doesn't solve getting it on their computers.
Any help here would be greatly appreciated.
Thanks!
--Rob

You could try uploading them to a joint dropbox account that they could then download them from....but how fast it is will depend on the internet connection available.

Best way to write objects in an xml and send the file?

hello all
i am making some objects, which i conver to xml using the XStream and then i am saving them to a file, call a function to send the file to a servant (basically i do this by sending bytes and this may be the problem). In the servant i take the data i write them to a file and then i read them again to make them objects...
i have the following code in the client
        XStream xstream = new XStream(new DomDriver());
        xstream.alias("Jobb", WorkflowFramework.Jobb.class);
     String xml = xstream.toXML(translation);
           FileWriter fw = new FileWriter("translation.xml");
           ObjectOutputStream out = xstream.createObjectOutputStream(fw);
           out.writeObject(new Jobb("ougk2", "Walnes",null));
           out.close();
           File file=new File("translation.xml");
           byte buffer[]=new byte[(int)file.length()];
           try {
                BufferedInputStream input=new BufferedInputStream(new FileInputStream("translation.xml"));
                input.read(buffer,0,buffer.length);
                input.close();
           } catch(Exception e) {
                  System.out.println("Error: "+e.getMessage());
                   e.printStackTrace();
           theRemoteObjRef.translationService(theCallbackObjectRef, buffer);i write the file and then i read it so as to have a buffer of bytes (there should be a better ways..)
the last line sends an objectRef (we dont care about that) and the file in bytes to a server (to be specific to a servant of a server)..
in the servant i do the following
public void translationService(TheCallbackInterface objRef, byte data[]){
          try{
                    File file = new File("translation2.xml");
                BufferedOutputStream output = new BufferedOutputStream(new FileOutputStream("translation2.xml"));
              output.write(data, 0, data.length);
              output.flush();
              output.close();
          } catch(Exception e) {
                    System.out.println("Error: " + e.getMessage());
                    e.printStackTrace();
           XStream xstream = new XStream(new DomDriver());
           try { // If an error occurs, go to the "catch" block
                FileReader fr2 = new FileReader("translation2.xml");
                ObjectInputStream in = xstream.createObjectInputStream(fr2);
                Jobb newJob = (Jobb)in.readObject();
               System.out.println(newJob.getObjServerRef());
               System.out.println(newJob.getForTranslation());
               System.out.println(newJob.getTranslated());
                }catch (Exception e) {
                     System.err.println("File input error"+e);
                Jobb testJob=new Jobb("ougka","mpougka","falalala");
                System.out.println(testJob.getObjServerRef());
                   System.out.println(testJob.getForTranslation());
                System.out.println(testJob.getTranslated());
the problem is that in the first group of System.out.println i get the error File input errorcom.thoughtworks.xstream.mapper.CannotResolveClassException: Jobb : Jobb
but the second group of printlns, prints the correct results...
do you know what is the problem? Why can i read the content of the file and get the objects from there?
do you have any suggestions of writing and reading the file in an another way or to send it by NOT using a bytes array?
do you have an alternative way of making the xml file?
many thanks!

Hi,
I would suggest to reconsider building of your document so that it doesn't contain duplicates if you don't need them. And it doesn't have much to do with DB XML then.
Also you could insert your document with duplicates and use XQuery Update facilities to delete undesirable nodes.
Vyacheslav

Best way to serialize large document to XML file

Similar Messages

Maybe you are looking for