Finding suitable indices for a large collection

Question

Hello,
I'm trying to find suitable indices for a large Xml collection.
I'm using a indexed node container. I added a large Xml document (about 450 MB). The schema of the document is like this:
<dbname>
<tablename1>
<row attribute1="attributevalue" attribut2="attributevalue" ... />
<row attribute2="attributevalue" attribut2="attributevalue" ... />
</tablename1>
<tablename2>
<row ......>
</dbname>
One table has got about 5 million entries. I'm not able to query this table, because my system runs out of memory (Windows begins to swap several gigabytes).
Typical queries are : "all rows of a table" and "all rows of a table with attribute=value"
I also tried using lazy queries, but it didn't get better.
The following indices I already added:
- edge-element-presence-none for dbname
- edge-element-presence-none for each tablename
- edge-element-presence-none for row
- edge-attribute-equality-string for attributes I'm trying to query
E.g. I'm trying to get all rows of a table. This is the query plan:
dbxml> queryplan collection("temp.dbxml")/dbname/tablename1/row
<XQuery>
<DbXmlNav>
<LookupIndex container="temp.dbxml">
<PresenceQP index="edge-element-presence-none" operation="eq" parent="tablename1" child="row"/>
</LookupIndex>
<Join type="parent-of-child" return="context_item">
<DbXmlNav>
<LookupIndex container="temp.dbxml">
<PresenceQP index="edge-element-presence-none" operation="eq" parent="dbname" child="tablename1"/>
</LookupIndex>
<Join type="parent-of-child" return="context_item">
<DbXmlNav>
<LookupIndex container="temp.dbxml">
<PresenceQP index="edge-element-presence-none" operation="eq" parent="root:http://www.sleepycat.com/2002/dbxml" child="dbname"/>
</LookupIndex>
<Join type="parent-of-child" return="context_item">
<QueryPlanFunction result="collection" container="temp.dbxml">
<OQPlan>n(P(edge-element-presence-none,=,tablename1.row),P(edge-element-presence-none,=,dbname.tablename1),P(edge-element-presence-none,=,root:http://www.sleepycat.com/2002/dbxml.dbname))</OQPlan>
</QueryPlanFunction>
</Join>
</DbXmlNav>
</Join>
</DbXmlNav>
</Join>
</DbXmlNav>
</XQuery>
When i execute this query (in eager mode), dbxml doesn't gives any response. The swap size is increased several gigabytes and the hard disk is working very hard. The same happens, when I use lazy mode and try to print out the results.
Could anybody please help me to improve the query performance?
Thanks,
Benjamin

Answer 1

Benjamin,
Using lazy evaluation should allow you to get an answer. The upcoming 2.4 release is easier on memory as well and should help.
That said, why are you even using an XML database if you are storing/querying what appears
to be relational data? Mapping relational database tables into a single XML document is going on only slow down access (due to the constraints XML imposes on the data that do not exist in a
relational system).
If you must do such things it's best to map at a finer granularity -- e.g. one table to one document or even one row per document.
Regards,
George