Indices configuration for XML document analysis (indexing time problems)

Question

Hi all,
I'm currently developing a tool for XML Document analysis using XQuery. We have a need to analyse the content of a large CMS dump, so I am adding all documents to a berkeley DB xml to be able to run xqueries against it.
In my last run I've been running to indexing speed problems, with single documents (typically 10-20 K in size) taking around 20 sec to be added to the database after 6000 documents (I've got around 20000 in total). The time needed for adding docs to the database drops with the number of documents.
I suspect my index configuration to be the reason for this performance drop. Indeed, I've been very generous with indexes, as we have to analyse the data and don't know the structure in advance.
Currently my index configuration includes:
- 2 default indicess: edge-element-presence-none and edge-attribute-presence-none to be able to speed up every possible xquery to analyse data patterns: ex. collection()//table//p[contains(.,'help')]
- 8 edge-attribute-substring-string indices on attributes we use often (id, value, name, ...)
- 1 edge-element-substring-string index on the root element of the xml documents to be able to speed up document searches: ex. collection()//page[contains(.,'help')]
So here my questions:
- Are there any possible performance optimisations in Database config (not index config)? I only set the following:
setTransactional(false);
envConf.setCacheSize(1024*64);
envConf.setCacheMax(1024*256);
- How can I test various index configuration on the fly? Are there any db tools that allow to set/remove indexes?
- Is my index config suspect? ;-)
Greetings,
Nils

Answer 1

Hi Nils,
The edge-element-substring-string index on the document element is almost certainly the cause of the slow document inserts - that's really not a good idea. Substring indexes are used to optimize "=", contains(), starts-with() and ends-with() when they are applied to the named element that has the substring index, so I don't think that index will do what you want it to.
John