Index-building strategy for multi-terabyte database

Question

Running 11g.
We have about 17 million XML files to load into a brand new database. They are to be indexed with a context index. After the 17 million records are imported, there will be future imports of approximately 30,000 records every two weeks, loaded in batch.
1. What is the best way to load the 17 million? Should they be loaded in chunks, say 1 M at a time, and then indexed? Or load them all, then index them all at once? (Based on preliminary tests the initial load will take 9 days and the indexing will take a little under 7 days.)
I vote for doing it in chunks, since the developers will want access to the system periodically during the data load and I want to be able to start and stop the process without causing a catastrophe.
But I have read that this can introduce fragmentation into the index. Is this really something to worry about, given that my chunks are so large? Would there be any real benefit from doing the entire index operation in one go?
2. After each of the bi-weekly 30,000-record imports, we will run CTX_DDL.SYNC_INDEX which I estimate will take about 20 minutes to run. Will this cause fragmentation over time? I have read that it is advisable to perform a full index rebuild occasionally but obviously we won't want to do that; that would take days and days. I guess the alternative is to run OPTIMIZE_INDEX
http://download.oracle.com/docs/cd/B28359_01/text.111/b28304/cddlpkg.htm#i998200
...any advice on how often to run this, considering the hugeness of my dataset? Fortunately the data will be read-only (apart from the bi-weekly load), so I'm thinking that there won't be very much fragmentation occurring at all. Am I correct?

Answer 1

There are two types of fragmentation, one during index creation and one during additional loads. The first can be minimised with some tweaking, the second should not be a big issue because there are only 26 additional loads in one year in.
1)
You will not have any issues loading the XML and index them using Oracle Text as long you use a sensible partitioning. Index the whole dataset in one go, the parallel clause works very well during indexing. Have a look at the initial memory allocation for every thread what is created, I found the default values in 10g and 9i are far too small. When you use 100 partitions it will create 100 Oracle text indexes, nothing to be scared off. More partitions you use, less index memory is required for each thread, but it adds to fragmentation.
You can reduce the initial indexing time and fragmentation in various ways:
Use parallel indexing and partitioning, I used 6 threads over 25 partitions and it reduced time to index 8 millions XML documents from 2 days to less than 12 hours (XML documents as clobs using a user_datastore).
Tweak indexing memory to your requirements, somehow it is more memory than CPU bound. More memory you use less fragmentation occurs and it will finish earlier.
Use a representative list of stop words. For this I use to query the DR$...I tables directly (query is out my head)
SELECT token_text, SUM(token_count) total
FROM DR$...I
WHERE UPPER(token_text) = tokent_text – remove XML tags
ORDER BY total DESC
Have a look at the top entries if you can include them into the stop word list; they will cause trouble later on when querying the index.
2) We found an index rebuild (drop/create) useful for following scenario, but this more a feeling than solid science
1 million XML records, daily loads of around 1 thousand records with many updates => quarterly rebuilds
8 million XML records, bi-weekly loads of around 20 thousand records with very few updates => once a year or so.