Efficient way of searching multiple xml files for multiple entries

As I�m quite new to using xml in java I can't figure out how to solve my problem.
I've got about 20 xml files, each about 500-1000kB big. each files contains about 500 questions, each with a unique ID.
A user had to be capable of entering any number of ',' separated id's, and the program needs to show the user the questions.
Using a SQL server this would be easy, but in this situation I can't. As this had to be a small program I can't add a 10MB jar file either, nor can I ask the users to install an additional program.
Creating a brute search will be easy, but searching 20MB of xml files multiple times will be hard even for a modern PC.
So my question is: What will be the most efficient way of searching these files?
Hope that someone will be kind enough to respond :)
Rick

I'd still go with a database. There are databases that are significantly more light-weight than MS SQL Server.
More concretely there are databases that run completely in memory. HSQLDB is one, Java DB (formerly Derby) is another one.
I'd parse the XML files once, add them to the database and query from there later on.
If even that is to complicated for you, then you could simply parse the XML files once and put the Questions into a HashMap with the ID as the key.

Similar Messages

Using multiple XML files for drill-down

I have an XML file which populates a graph showing months of the year. I have 12 other XML files which have data for each day of the month running from 1 - 31.
What I would like to do is use the 12 XML files to drill down for each of the months on my graph, but am struggling to get it to work.
The XML that populates the graph is in the following format :
<?xml version="1.0" encoding="utf-8"?>
<items>
     <item month="Jan" value="536102" />
     <item month="Feb" value="484570" />
     <item month="Mar" value="155840" />
and the other XML files are in the following format:
<?xml version="1.0" encoding="utf-8"?>
<items>
     <item Timestamp="24/04/2010" Value="4178" />
     <item Timestamp="25/04/2010" Value="8075" />
     <item Timestamp="26/04/2010" Value="14611" />
Would it be simpler to combine all the XML into one file?
Sorry that this is a bit vague, but if you could help me that would be appreciated. Let me know if you require any more information.
Thanks.

You can only import a single XML file. You may have an xml file that has multiple occurances of data but it must be a single xml file.
So you woudl have to find a way to combine all of those xml files into one then do the import.
Paul

Multiple XML files for one application

I have an application I wrote for my company and they all use the same application, but depending on there department they need to access a different XML file on the server. How do i code it so that the user has to input a code the first time they open the application in there phone and then it will know what XML file to pull from. Any help in how to structure this would be great. I am using the HTTP service call.

Each logger can have its own file I think. Check the log4j documentation.

One file for multiple IDOCs

Hi,
Need help on IDOC files created on application server.
For IDOCs of EDI type generally one file is created on application server for one idoc. Is there any way to create a common file for multiple IDOCs that are created in a batch job?
Thanking in anticipation
Sudhir Bhate

Hi Sudhir;
Yes, it is a setting on the outbound parameters of the partner profile. In the section for "output mode," select "Collect IDocs." This link to help.sap.com describes the procedure:
http://help.sap.com/saphelp_nw04/helpdata/en/dc/6b812a43d711d1893e0000e8323c4f/content.htm
Cheers,
John

Efficient searching in a large XML file for specific elements

Hi
How can I search in a large XML file for a specific element efficiently (fast and memory savvy?) I have a large (approximately 32MB with about 140,000 main elements) XML file and I have to search through it for specific elements. What stable and production-ready open source tools are available for such tasks? I think PDOM is a solution but I can't find any well-known and stable implementations on the web.
Thanks in advance,
Behrang Saeedzadeh.

The problem with DOM parsers is that the whole document needs to be parsed!
So with large documents this uses up a lot of memory.
I suggest you look at sometthing like a pull parser (Piccolo or MPX1) which is a fast parser that is program driven and not event driven like SAX. This has the advantage of not needing to remember your state between events.
I have used Piccolo to extract events from large xml based log files.
Carl.

Is there a way to import large XML files into HANA efficiently are their any data services provided to do this?

1. Is there a way to import large XML files into HANA efficiently?
2. Will it process it node by node or the entire file at a time?
3. Are there any data services provided to do this?
This for a project use case i also have an requirement to process bulk XML files, suggest me to accomplish this task

Hi Patrick,
I am addressing the similar issue. "Getting data from huge XMLs into Hana."
Using Odata services can we handle huge data (i.e create schema/load into Hana) On-the-fly ?
In my scenario,
I get a folder of different complex XML files which are to be loaded into Hana database.
Then I gotta transform & cleanse the data.
Can I use oData services to transform and cleanse the data ?
If so, how can I create oData services dynamically ?
Any help is highly appreciated.
Thank you.
Regards,
Alekhya

Is there any way to upload Tariff Code (with multiple XML files) from application server?

Hi All,
Is there any way to upload Tariff Code (with multiple XML files) from application server?. Its urgent.
Regards,
Jatin

Hi Jatin,
Yes, of course you can upload multiple files for tariff codes.
This can be done by the below path:-
SAP GTS Cockpit(tcode-/sapsll/menu_legal)-->Customs Management-->Classification-->Classification Master Data-->Upload Tariff Code Numbers from XML file(tocde- /SAPSLL/LLNS_UPL101).
In the above area after browsing and choosing the first file, please select multiple check box to choose more files as well. Then you can further select your application server and upload all those files in one go.
PS:- Although, we have an option to upload multiple such files but actually we should avoid multiple file uploads due to various reasons. Hence, please take utmost care during such procedure.
Regards,
Aman

Multiple log4j.xml files for single application.

Hi All,
How to use multiple log4j.xml files for single application?
I do have a pluggable application modules. ie, If I add a jar I will get some functionalities. Like that i do have many jars.
Log4j.xml also will be present in that jar only. So if I add multiple jar files like this, I will get multiple log4j.xml files. What should I do?

Each logger can have its own file I think. Check the log4j documentation.

HR - XI - External System: one IDOC to multiple XML Files

Hi,
I have a scenario where I need, out of one HRMD_A07 IDOC send multiple XML files:
The IDOC and the XML files can the Master data for many employee (IDOC has max of 200 as of SAP, external system, max of 1000).
The IDOC will be generated with change pointers with all the needed filters (I can add/remove if needed).
My goal is to generate 3 types of XML files:
- 1 for the persons (containing basic info like name, firstname, persnr,etc)
- 1 for the various unit (with basic info of unit like number, description & name)
- 1 for the links between persons and units (this person is linked with that unit)
The structures of the 3 XML files are know and can be imported via XSD definition.
Do I need to use BPM for doing so or is there a way to do that with simple mapping?
If BPM is needed, as I new to that, if you have a link to a begginer guide, fell free to send it
Cheers,
greg

hi,
For your scenario there is no need of BPM.
In message mapping select the message tag.
In the target add 3 message types.
Just map the fields.Go to Interface mapping add the 3 message in the target.
Get the mapping.
In ID sender agreement,Receiver Determination is same.
In Interface Determination select extended.Get the mapping.
Create 3 Receiver agreement for each Receiver.
Regards,
Prakasu

Loading multiple XML files into one XMLList

I am working on an app where I will load a list of recipes. Each recipe is an XML file. I would have an XML file that lists the filenames that need to be loaded. I am assuming in can use one HTTPService and loop through this list, and set the url and do myService.send() until all the xml files have been loaded and concatenated into one xml list. True?
My main question here is how do I load each XML file and add it to the same XMLList?
Assume each XML file's root tag is <recipe>. I want to create an XMLList with the root <recipes> with each <recipe> as a first child.
Thanks.

Why don't you use the Data centric workflows which will help you to deal with multiple XML files as AS objects? Something like this
1. Create a HTTPService using Data menu -> Connect to HTTPService
2. create a operation getRecipe and enter your URL example URL: http://myserver/recipe?name=recipename
3. Use Configure Return Type to tell that this method returns a Recipe class
4. Use DataBinding to bind the result of this to a DataGrid
Call the following in a loop and
recipeName = myRecipes+ // All of your recipe names, Recipe is a ArrayCollecition
Recipe.setAt(i) = Svc1.getRecipe(recipeName);
At the end you would have a Recipe[]
This is one of the ways you could work instead of having a XMLListCollection and use e4x etc. to go through your collection.
Let us know if this helps, or you are looking for something else.
Thanks
-Sunil

ESB/File Adapter - XML files containing multiple messages

Hi,
In all the examples on file adapters I read, if files contain multiple messages, it always concerns non-XML files, such as CSV files.
In our case, we have an XML file containing multiple messages, which we want to process separately (not in a batch). We selected "Files contain Multiple Messages" and set "Publish Messages in Batches of" to 1.
However, the OC4J log files show the following error:
ORABPEL-12505
Payload Record Element is not DOM source.
The Resource Adapter sent a Message to the Adapter Framework which could not be converted to a org.w3c.dom.Element.
Anyone knows whether it's possible to do this for XML files?
Regards, Ronald

Maybe I need to give a little bit more background info.
Ideally, one would only read/pick-up small XML documents in which every XML document forms a single message. In that way they can be processed individually.
However, in our case an external party supplies multiple messages in a single batch-file, which is in XML format. I want to "work" on individual messages as soon as possible and not put a huge batch file through our ESB and BPEL processes. Unfortunately we can not influence the way the XML file is supplied, since we are not the only subscriber to it.
So yes, we can use XPath to extract all individual messages from the XML batch-file and start a ESB process instance for each individual message. But that would require the creation of another ESB or BPEL process which only task is to "chop up" the batch file and start the original ESB process for each message.
I was hoping that the batch option in the File adapter could also do this for XML content and not only for e.g. CSV content. That way it will not require an additional process and manual coding.
Can anyone confirm this is not supported in ESB?
Regards,
Ronald
Message was edited by:
Ronald van Luttikhuizen

1 xml file to multiple xml files with 200 records in each

Hello Experts,
I have below scenario.
Please suggest me which might be the best approch to work on.
1) XML file to XML file
I will have 1 pretty huge XML file which i need to break in multiple XML files with 200 records each.
1) first approch i can have bpm in which i can split the file according to my requirement.
2) Second approch i can create 2 scenarios in which 1st scenario will pick up XML file and create multiple flat files with File content conversion. Second scenario will pick up all these flat files and create XML files.
2) XML file to XML file
Or i can have multiple files with 1 record per file and i need to merge in multiple XML files with 200 records in each.
So its kind of 1:N or M:N scenarios.
Please tell me which is might be better performance and design wise.
Or if you have any idea in any other way i can do this, then please reply as soon as possbile.
Please tell me if you have OS command for the same or some script to run or anything which i can implement.
Thanks,
Hetal

what is your senario? is it File to File?
You can use multi mapping concept without BPM. You can handle the 200 records per message logic in the multimapping.
Regards,
Praveen Gujjeti.

Insert Insert XML file into multiple records in Oracle Database

I would like to find out if it is possible to insert a single XML file into multiple records or tuples in a Oracle database table. I have a single XML file which is at multiple levels. The meta data for the levels are common and each level can have meta data of their own in addition. I do not have any meta data field which will uniquely determine whether the data belongs to Root level, Intermediate level or at the document level. Is there any way I can determine which level the meta data belongs to and thereby make a corresponding entry into the database table tuple? For example I could have an attribute called level which is going to be present only in the database table and not in the XML file. If level=1 then it corresponds to "Root" meta data, if level=2 then it corresponds to "Intermediate" level and if level=3 then it corresponds to meta data at document level. I need a way to calculate the value for level from the XML file and thereby insert the meta data element into a tuple in a single table in Oracle.

Hi,
extract your xml and then you can use insert all clause.
here's very small example on 10.2.0.1.0
SQL> create table table1(id number,val varchar2(10));
Table created.
SQL> create table table2(id number,val varchar2(10));
Table created.
SQL> insert all
2 into table1 values(id,val)
3 into table2 values(id2,val2)
4 select extractValue(x.col,'/a/id1') id
5        ,extractValue(x.col,'/a/value') val
6        ,extractValue(x.col,'/a/value2') val2
7        ,extractValue(x.col,'/a/id2') id2
8 from (select xmltype('<a><id1>1</id1><value>a</value><id2>2</id2><value2>b</value2></a>') col from dual) x;
2 rows created.
SQL> select * from table1;
        ID VAL
         1 a
SQL> select * from table2;
        ID VAL
         2 b                                                                    Ants

The most efficient way to search a large String

Hi All,
2 Quick Questions
QUESTION 1:
I have about 50 String keywords -- I would like to use to search a big String object (between 300-3000 characters)
Is the most efficient way to search it for my keywords like this ?
if(myBigString.indexOf("string1")!=1 || myBigString.indexOf("string2")!=1 || myBigString.indexOf("string1")!=1 and so on for 50 strings.)
System.out.println("it was found");
QUESTION 2:
Can someone help me out with a regular expression search of phone number in the format NNN-NNN-NNNN
I would like it to return all instances of that pattern found on the page .
I have done regular expressions, in javascript in vbscript but I have never done regular expressions in java.
Thanks

Answer 2:
If you have the option of using Java 1.4, have a look at the new regular expressions library... whose package name I forget :-/ There have been articles published on it, both at JavaWorld and IBM's developerWorks.
If you can't use Java 1.4, have a look at the jakarta regular expression projects, of which I think there are two (ORO and Perl-like, off the top of my head)
http://jakarta.apache.org/
Answer 1:
If you have n search terms, and are searching through a string of length l (the haystack, as in looking for a needle in a haystack), then searching for each term in turn will take time O(n*l). In particular, it will take longer the more terms you add (in a linear fashion, assuming the haystack stays the same length)
If this is sufficient, then do it! The simplest solution is (almost) always the easiest to maintain.
An alternative is to create a finite state machine that defines the search terms (Or multiple parallel finite state machines would probably be easier). You can then loop over the haystack string a single time to find every search term at once. Such an algorithm will take O(n*k) time to construct the finite state information (given an average search term length of k), and then O(l) for the search. For a large number of search terms, or a very large search string, this method will be faster than the naive method.
One example of a state-search for strings is the Boyer-Moore algorithm.
http://www-igm.univ-mlv.fr/~lecroq/string/tunedbm.html
Regards, and have fun,
-Troy

Creating multiple XML files from Schema

We are extracting data from Oracle DB into XML using ODI. We are successful in generating the XML file, using command "create xmlfile <file name> from schema <Schema Name>, but the entire data comes in Single file.
Our database schema is relatively Huge and we would like to generate multiple XML files. We want to generate one file per particular number of records. Can anyone help to do this.

one solution i can think of is to use SQLX operator instead of dbms_xmlgen.
here is a sample example.
declare
l_xmltype xmltype;
l_deptno emp.deptno%type;
begin
for i in (select * from emp order by deptno)
loop
    select xmlconcat(
                xmlelement("ename", i.ename)
               ,xmlelement("sal", i.sal)
               ,xmlelement("detpno", i.deptno))
      into l_xmltype from dual;
    dbms_output.put_line(l_xmltype.GetClobVal());
end loop;
end;
/Now here you can open the query once, keep writing to the file till the deptno
is same, when the deptno changes, close the file and open a new file with new
deptno and start writing.
Note : in this way you will have to add the xmlprolog manually to each of the file which should not be an issue. after opening the file add the prolog string manually.
Hope this helps.

Efficient way of searching multiple xml files for multiple entries

Similar Messages

Maybe you are looking for