Using XML to import/export large datasets

Hi,
I have a large customer who is using serveral aplications which
exchange data. Now they use MQUE Series to put some data on the
line. Each external application has his own interface.
We would like to use one interface which uses XML and depending
on what is in de xml-document, writes it in different Oracle
databases.
On a daily bases there must be created 500.000 records in de
database, so performance is a hot issue.
What are the possibilities and needed configuration (Oracle
8...i) and how reliable it is going to be ?
Suppose we want to use as much as pl/sql as possible.
Thanks for any feedback,
Ed
[email protected]
null

Ed Somers (guest) wrote:
: Hi,
: I have a large customer who is using serveral aplications
which
: exchange data. Now they use MQUE Series to put some data on
the
: line. Each external application has his own interface.
: We would like to use one interface which uses XML and
depending
: on what is in de xml-document, writes it in different Oracle
: databases.
: On a daily bases there must be created 500.000 records in de
: database, so performance is a hot issue.
: What are the possibilities and needed configuration (Oracle
: 8...i) and how reliable it is going to be ?
: Suppose we want to use as much as pl/sql as possible.
: Thanks for any feedback,
: Ed
: [email protected]
You should check out the Oracle Integration Server which was
just announced at Openworld. It is ax XML Messaging Hub that
incorporates workflow and has an adapter SDK to connect to your
applications. It uses JMS as the envelope and XML as the payload.
Oracle XML Team
http://technet.oracle.com
Oracle Technology Network
null

Similar Messages

[Forum FAQ] How do i use xml stored in database as dataset in SQL Server Reporting Services?

Introduction
There is a scenario that users want to create SSRS report, the xml used to retrieve data for the report is stored in table of database. Since the data source is not of XML type, when we create data source for the report, we could not select XML as Data Source
type. Is there a way that we can create a dataset that is based on data of XML type, and retrieve report data from the dataset?
Solution
In this article, I will demonstrate how to use xml stored in database as dataset in SSRS reports.
Supposing the original xml stored in database is like below:
<Customers>
<Customer ID="11">
<FirstName>Bobby</FirstName>
<LastName>Moore</LastName>
</Customer>
<Customer ID="20">
<FirstName>Crystal</FirstName>
<LastName>Hu</LastName>
</Customer>
</Customers>
Now we can create an SSRS report and use the data of xml type as dataset by following steps:
In database, create a stored procedure to retrieve the data for the report in SQL Server Management Studio (SSMS) with the following query:
CREATE PROCEDURE xml_report
AS
DECLARE @xmlDoc XML;
SELECT @xmlDoc = xmlVal FROM xmlTbl WHERE id=1;
SELECT T.c.value('(@ID)','int') AS ID,
T.c.value('(FirstName[1])','varchar(99)') AS firstName,
T.c.value('(LastName[1])','varchar(99)') AS lastName
FROM
@xmlDoc.nodes('/Customers/Customer') T(c)
GO
P.S. This is an example for a given structured XML, to retrieve node values from different structured XMLs, you can reference here.
Click Start, point to All Programs, point to Microsoft SQL Server, and then click Business Intelligence Development Studio (BIDS) OR SQL Server Data Tools (SSDT). If this is the first time we have opened SQL Server Data Tools, click Business Intelligence
Settings for the default environment settings.
On the File menu, point to New, and then click Project.
In the Project Types list, click Business Intelligence Projects.
In the Templates list, click Report Server Project.
In Name, type project name.
Click OK to create the project.
In Solution Explorer, right-click Reports, point to Add, and click New Item.
In the Add New Item dialog box, under Templates, click Report.
In Name, type report name and then click Add.
In the Report Data pane, right-click Data Sources and click Add Data Source.
For an embedded data source, verify that Embedded connection is selected. From the Type drop-down list, select a data source type; for example, Microsoft SQL Server or OLE DB. Type the connection string directly or click Edit to open the Connection Properties
dialog box and select Server name and database name from the drop down list.
For a shared data source, verify that Use shared data source reference is selected, then select a data source from the drop down list.
Right-click DataSets and click Add Dataset, type a name for the dataset or accept the default name, then check Use a dataset embedded in my report. Select the name of an existing Data source from drop down list. Set Query type to StoredProcedure, then select
xml_report from drop down list.
In the Toolbox, click Table, and then click on the design surface.
In the Report Data pane, expand the dataset we created above to display the fields.
Drag the fields from the dataset to the cells of the table.
Applies to
Reporting Services 2008
Reporting Services 2008 R2
Reporting Services 2012
Reporting Services 2014
Please click to vote if the post helps you. This can be beneficial to other community members reading the thread.

I have near about 30 matrics. so I need a paging. I did every thing as per this post and it really works for me.
I have total four columns. On one page it should show three and the remaining one will be moved to next page.
Problem occurs when in my first row i have 3 columns and in next page if I have one columns then it show proper on first page but on second page also it gives me three columns insted of one. the first column data is exactly what I have but in remaining two
columns it shows some garbage data.
I have data like below.
Metric ColumnNo RowNo
1 1
1
2 2
1
3 3
1
4 1
2
so while grouping i have a row parent group on RowNo and Column group on ColumnNo.
can anyone please advice on this.

Import/Export Large Complex XMl files(700MB) to MySQL tables

Am creating an application that will be mporting large and complex XML file in the range of 500-1000MB size. An xml ile wil have 5-6 levels of xml tags where the first tag will be the client account configuration e.g name, location, address, the second will be orders made, the third will be order items, fourth will be item defination...etc. The following is a sample structure of the file.
<mycompany_sales> <customer name="me"> <location country="Kenya"> <town>nairobi</toen> <order id="20000" date="2008-09-08"> <item id="4000"> <stock>2000</stock> <buying>300.0</buying> </item> ........ <total>5000</total> </order> .............. </location> </customer> ........... </mycompany_sales>
Because of the size, if i process this file using DOM it will result to out of memory error. I have tried parsing with SAX but it takes about 18 hours to process which is not ideal. If i import this data to Microsoft Access it takes about 10 minutes to import including it's relationship. i.e. creates the six tables and the foregin key relations.
I tried using XML-DBMS package but since it relies on DOM, it cant process. Does anyone know of a package that would help me import this data to any JDBC databse and also recreate the same structure of xml from the jdbc db. It would also be ideal if it can create sample map file during import and export.
Thanks in advance,
Simon K.

kagara wrote:
I tried using XML-DBMS package but since it relies on DOM, it cant process. Does anyone know of a package that would help me import this data to any JDBC databse and also recreate the same structure of xml from the jdbc db. It would also be ideal if it can create sample map file during import and export.So you are parsing the XML, extracting the schema, using jdbc to create it, then parsing the data and using jdbc to populate it.
I suggest that you just create SQL directly. Or perhaps a import file if MySQL has it.
Less impact on the database, easier to test, easier to verify, and much faster to actually apply the data.

ReportMigration Using LCM & Simple Import & Export.

Hi,
I just want to know the difference between these two.. migrating the reports using LCM or by a simple import & export.
Will LCM even migrate the user security which is associated with the report??
Can anybody throw some light on this.
Thanks
Jagadesh R K.

Yes, LCM can be used to export / import Reports / Application provisioning.
For more details, refer the below documents
EPM 11.1.1.x
http://download.oracle.com/docs/cd/E12825_01/epm.111/epm_lifecycle_management.pdf
System 9.3.1
http://download.oracle.com/docs/cd/E10530_01/doc/epm.931/hss_alcm_util.pdf
Hope this helps.
Regards,
Manmohan Sharma

Using DB2's import/export utilities in Java program

DB2 database has build in utility called export, throgh which we can export the output of a Query directly onto a Lotus 123 file. But I beleive that , to invoke such DB2 utilities throgh a Java Program, we need to have APIs.
My question is ,are such APIs available ?
If so, what are the APIs necessary for calling this EXPORT and IMPORT utilities of DB2 ?

Well, If you have to use Runtime.exec() and provide these as command lines, then the best option is to write a shell script( my Java application is on AIX platform) and then call the shell script through exec().
But Shell scripts don't run well with Runtime.exec() right ?
B'Cause, I tried it but the application did'nt throw up any exception. No message was flashed either. The process was just hanging.....so do you know a better method to actually run a shell script through a Java program ?

Using bash to import/export from qt

i have an operation i perform repeatedly and would like to write
a bash script to do it automatically.
what i'd like to do is:
1. open an image sequence in qtplayer
2. file--->export
3. set options (like custom size, what codec)
4. save the export as the same filename as the first file of the image sequence
can this be done via bash?
can anyone point me to some info on passing parameters to qtplayer via bash?
thanks,
BabaG

Have you tried using Applications -> Automator -> Record and then let Automator record steps you want to do. If that works, it would be the easiest way to automate this.
If you want to invoke the Automator script you can use the open command from within bash
If Automator -> Record does not work for you needs, then an Applescript is the next thing you should try. If you want to invoke this from bash use the osascript command.
Applescript questions would be best asked in the Apple Applescript forum <http://discussions.apple.com/forum.jspa?forumID=724>
If you want/need more bash shell script questions answered, the Apple Unix forum would be the best bet <http://discussions.apple.com/forum.jspa?forumID=735>
And while I'm mentioning the open command, you can start *QuickTime Player* with a specific file usng open
open -a /Applications/QuickTime Player.app file.jpg
Where file.jpg could be any file that *QuickTime Player* knows how to handle.
Steps 2, 3, & 4 are operations that would be better handled via Automator or Applescript.

Thread death and xml files when analyzing large datasets

i analyze large data sets. for example:
4000 persons that each have a time series with 8,000 data points. i do this linearly, so (i think) i am only running the
*public static void main(String[] args) { .... }*
for(Person person : PersonSet) { // there are 4,000 people
List<Data> data = DataSink.getData(); // each List has a size of 8,000
analyzer.analyze(person, data); // i do not start any threads in this method
}i've run into, and over come, java.lang.OutOfMemoryError issues.
now, my program runs fine for over two minutes, then it crashes and shows this error:
C:\Documents and Settings\David\My Documents\NetBeansProjects\Test\nbproject\build-impl.xml:419: The following error occurred while executing this line:
C:\Documents and Settings\David\My Documents\NetBeansProjects\Test\nbproject\build-impl.xml:286: java.lang.ThreadDeath
this is how i expand memory:
-Xms1250m -Xmx1250m
i cannot understand how run-time memory management relates to xml files.?
my IDE is NetBeans 6.1

i've never dealt with ANT scripts, so this is like my first "Hello World" programming with ANT.
yet,
i can't understand why the jvm would say that the origin of a crash in a program thats be running for over 2-minutes
would be in an ANT file?
#more build-impl.xml
    <target name="-init-macrodef-java">
        <macrodef name="java" uri="http://www.netbeans.org/ns/j2se-project/1">
            <attribute default="${main.class}" name="classname"/>
            <element name="customize" optional="true"/>
            <sequential>
// line #286                <java classname="@{classname}" dir="${work.dir}" fork="true">   // <--- line #286
                    <jvmarg line="${run.jvmargs}"/>
                    <classpath>
                        <path path="${run.classpath}"/>
                    </classpath>
                    <syspropertyset>
                        <propertyref prefix="run-sys-prop."/>
                        <mapper from="run-sys-prop.*" to="*" type="glob"/>
                    </syspropertyset>
                    <customize/>
                </java>
            </sequential>
        </macrodef>
    </target>
    
    <target depends="init,compile" description="Run a main class." name="run">
// line #419        <j2seproject1:java>     // <-- line #419
            <customize>
                <arg line="${application.args}"/>
            </customize>
        </j2seproject1:java>
    </target>
    <target name="-do-not-recompile">
        <property name="javac.includes.binary" value=""/>
    </target>
    <target depends="init,-do-not-recompile,compile-single" name="run-single">
        <fail unless="run.class">Must select one file in the IDE or set run.class</fail>
        <j2seproject1:java classname="${run.class}"/>
    </target>
............................thanks

Using automator to import/export from qt

i have an operation i perform repeatedly and would like to write
an automator script to do it automatically.
what i'd like to do is:
1. open an image sequence in qtplayer
2. file--->export
3. set options (like custom size, what codec)
4. save the export as the same filename as the first file of the image sequence
can this be done via automator?
can anyone point me to some info on passing parameters to qtplayer via automator?
thanks,
BabaG

I would dearly like to know if there is a solution to this as well.
Trying to bash out a Cleaner watch folder template where
1) OSX will constantly monitor a drop folder
2) wrap any image sequences up as Quicktimes
3) send the Quicktime to a new folder for Cleaner to take over

OIA - Importing nested entitlements using XML file

Hi,
I am looking for some information on importing nested entitlements (like RACF information) into OIA. From what I gather sofar, looks like I have to use XML to import nested entitlements. But sofar, I have no luck finding any documentation on how to go about it.
If anybody has any information that you can share, that would be great.
Thank you
-Kamal

If you're saying that you have Reader 9 and Acrobat Pro 7 on the same machine, please be aware that this is not a recommended or supported configuration. Although it's possible, people have reported a number of problems with such a setup.
George

Attunity connectors for Oracle in Import Export Wizard in SQL Server 2008 R2

Is there a way we can see the Attunity connectors drivers in the Import/Export Wizard (64 bit) for SQL Server 2008 R2?
Although I made it work for SSIS, I would need these drivers in the Import/Export wizard so as to automate it for numerous number of tables which I want to migrate.
Can the Attunity connectors for Oracle be used in the Import/Export wizard? If so please let me know.
Regards,
Ashutosh.
Ashutosh.

I have 100 tables to migrate. Creating a data flow for each table is tedious and that's why I was looking out for a way to do it through import export wizard so that I don't have to create a separate data source and destination for each table in the Data
flow Task.
Is there a way to loop through all tables and transfer data in SSIS without having multiple sources and destinations created for each table? This also involves a bit of transformation as well.
Regards,
Ashutosh.

CSS Import/Export Utility

I'm trying to export users and groups along with their provisions using the CSS Import/Export utility, and I've configured my importexport.properties file to include export.provisioning.all=true, but the exported file does not include any provisioning. Is there a way to fix this?

I agree with Seb. I could not get the 9.2.0.1 utility to work. It was exporting only native groups and the members in each group and not any provisioning info.
Once I installed the 9.3.1 version of the utility, it all worked fine. Make sure you install JAVA JDK 1.5 as the utility needs that version. Just install in same path as the cssimportexport utility so that it doesn't conflict with any earlier versions of java (we had 1.4.2) on that server.
Jeff

STSADM Import/Export

Hello,
I am using SharePoint 2007.
I am trying to migrate some subsites from one site collection to other using STS ADM Import/Export command. while migrating I am facing some issues related to document versions and List data.
After migration some data from document library and lists, some data is missing. I am not sure why this is happening.
what should I do to avoid this situation?
does anyone face similar issue?
Is there any other way to migrate data from one site collection to other without any loss in data?
Thanks in advance.

I tend to use this: http://spdeploymentwizard.codeplex.com/
Scott Brickey
MCTS, MCPD, MCITP
www.sbrickey.com
Strategic Data Systems - for all your SharePoint needs

Automate Outlook Contact import/export?

I am in the process of developing a contact management system
for my company and have built coldfusion pages that allow the user
to import their personal contacts from outlook, as well as export
contacts from the system into their MS Outlook contacts.
The process is fairly simple on the users (in my mind) but it
requires using Outlook's import/export function, which is about 6
or 7 steps. My users would of course like to have this process
simplified.
Is it possible to have a button on one of my pages that
automates the entire process? i.e. creates the csv file (or
whatever other format is needed) and then automatically pushes that
data into outlook (and vice versa) ?
Any thoughts or suggestions are greatly appreciated!

If your Outlook users are using Exchange you might look into
ColdFusion's Exchange features.
http://livedocs.adobe.com/coldfusion/8/htmldocs/help.html?content=cfexchange_1.html#112045 8

Using XML Import/Export in Transport Connection

Hi, I was investigating the feasibility of copying a query definition from one BW system to another using the XML Import/Export functionality in Transport Connection. This is not to replace our current transport process across the landscape (Dev -> QA -> Prod), but merely just looking at possibilities for end users who develop queries in QA & then recreate them in Prod.
My question is: has anyone worked with the XML Export/Import for types Queries & Query Elements? I was trying this out between a Dev and Sandbox system and only had limited success. I was able to take a small query and perform the Export to a local .xml file without much difficulty. But when I try to do the Import, my query never shows up. The Import function shows a green light, but I get a couple of error messages on the Import such as the following:
==========================================================
SAP object 3WROG4HZ3NKP1PIYHCR1H24CQ () cannot be imported
Diagnosis
You attempted to import SAP object 3WROG4HZ3NKP1PIYHCR1H24CQ of type into the system via the XMI interface. However, you are not allowed to import SAP objects.
System response
The object was ignored during the import.
Note: It can be that the import is incomplete when the required SAP objects are still not active in the system.
Procedure
Install the specified objects from Business Content. Only import the metadata afterwards.
=========================================================
The portion returned for Saving and Activating returns green lights. The two sample objects mentioned appear to just be custom InfoObjects used in my query & don't look to be any different than others that work OK.
Just curious if anyone else has worked with this and could help advise.
Thanks... Jody

This is an old subject, but I think XML import only works for datamodeling objects like InfoObjects, InfoCubes, and ODS objects. If you try to export a query, and open the XML file in a text editor, there is not enough information (in my opinion) to build the query. As an example you don't see how key figures are to be displayed, filter or free characteristics.
However, when an InfoObject is collected, you get all the attributes, texts in all languages.
If anyone has more insight, please enlighten us.
-John

Smart export/import - unable to import project that uses 'xml' technology

Hello,
I have smart exported project and this project uses technology 'XML'. When I'm trying to import it, there are following issues (screen) - no fix available to choose:
http://screener.tk/f/o/f/z6nRM.png
So, it seems technology 'XML' is not recognized during import, because it has different ID. On repository I'm trying to import to, XML has ID 27999. So, I manually edited SmartExport.xml file and changed all occurences of 1189 to correct ID: 27999 and tried again to import. After this change, there are no issues. But just after I start import, following error message is thrown and import fails:
http://screener.tk/f/o/f/wf8R0.png
The same issue when I'm trying to import any project that uses any not built-in technology. After importing technology on destination server, ID doesn't match. And if I change ID in SmartExport.xml, no issues but import fails due to error similar as above.
So, what is the correct way to perform succesfull import/export in this cases and how to fix "dirrerent origin" errors? Do you have any idea?

PS - have found other posts indicating that clips smaller than 2s or sometimes 5s, or "short files" can cause this. Modern style editing often uses short takes ! Good grief I cannot believe Apple. Well I deleted a half a dozen short sections and can export, but now of course the video is a ruined piiece of junk and I need to re-do the whole thing, the sound etc. which is basically taking as much time as the original. And each time I re-do it I risk again this lovely error -50 and again trying to figure out what thing bugs it via trial and error instead of a REASONABLE ERROR MESSAGE POINTING TO THE CLIP IT CAN'T PROCESS. What a mess. I HATE this iMovie application - full of BUGS BUGS BUGS which Apple will not fix obviously, since I had this product for a few years and see just hundreds of hits on Google about this error with disappointed users. Such junk I cannot believe I paid money for it and Apple does not support it with fixes !!!
If anyone knows of a GOOD reasonably priced video editing program NOT from APPLE I am still looking for suggestions. I want to do more video in future, but obviously NOT with iMovie !!!

Using XML to import/export large datasets

Similar Messages

Maybe you are looking for