10gR2 Incremental aggregation performance

Question

Hi, I've seen the Oracle presentation about 10gR2 where it shows that incremental data loads perform remarkably faster than they did in 10gR1, etc. However, I have not been able to repeat this performance on a small test cube. Could someone from Oracle please jump in and let me know if there are additional settings, design considerations, etc. needed to see these performance increases?
The cube I'm testing this on (borrowed from Mark Rittman) has the following setup:
qty_shipped <channel <time customer product promotion>>
qty_ordered <channel <time customer product promotion>>
I am using a global composite, but not compression. Also I am not doing a skip-level aggregation - I'm fully aggregating all dimensions.
When I load and aggregate the entire data (11,429 input records), I see the following performance:
Data load time: 1 sec
Aggregation time: 33 sec
Basically, then I delete the database, recreate it from the XML template, and load it in two pieces: an "initial" load (containing 11,411 records) and an "incremental load" (containing 18 records). These tables are populated based on time - the initial load has all history except for the last two days worth of data, and the incremental load contains the last two days worth.
Here is the performance I see:
"Initial" load - solving the exact same way as the "full" load above except using the "initial" load table (11,411 recs), and then doing an "aggregate full cube":
Data load time: 1 sec
Aggregation time: 40 sec
*** Note: for some reason, aggregating the "initial" load table is always somewhat slower than the one I build using the "full" dataset - not sure why! Doing the exact same steps, starting with clean cube...only diff would appear to be that the "initial" table has 18 less records than the full one. Any ideas?
"Incremental" load - here I load just the 18 records for the last two days worth of data. I repoint the cube mappings to the "incremental" load table, then load and aggregate using the "aggregate only for incoming values" method. I would expect this solve to be so fast I can barely measure it, but it isn't:
Data load time: 1 sec
Aggregation time: 17 sec
Obviously the incremental load is somewhat faster than doing a full load - but on the other hand, its only loading 18 out of 11,000+ values. I would expect the whole thing to finish in under a second.
Could someone please shed some light on what is going on?
Thanks!
Scott

Answer 1

2. Are there any good reference materials on how to
set up partitioning for OLAP? I'm intrigued by your
comment re: parallel builds, etc. but I no nothing
about how to effectively set up and use local
composites, etc.I don't know what reference materials are available
for this sort of thing.
Also, I neglected to point out in the last message
that, AFAIK, you can only use global composites
if you are NOT using compression. (At the OLAP DML
level, you can't have more than one partition
dimensioned by the same compressed composite
anymore than you can have two different variables
dimensioned by the same compressed composite.)
To expand a bit on the issue of parallel
build (actually, parallel load / build):
AWM lets you use the job queue to do your
refresh in multiple sessions simultaneously.
These sessions use multiwriter mode, and
each one basically ACQUIRE's a partition,
loads and builds it, and then UPDATE's it;
then, it moves on to another partition. If two
partitions share a composite, then two sessions
can't ACQUIRE it simultaneously, so global
composites can interfere with parallelism. (If you
are refreshing more than one cube at a time, however,
it's possible that AWM may do one cube in one session
at the same time as it does another cube in a different
session - I'm not sure about this.)
>
3. How does partitioning TIME work when time is a
dense (fastest varying) dimension of a variable? My
understanding (under Express) is that the fastest
varying dimensions are all grouped together on a
single database "page". If you partition on TIME, how
will it break up those pages? Does a "page" thenSuppose that you have a cube dimensioned by
TIME, PROD, and GEOG, and TIME is both dense
AND the fastest varying dimension. (TIME is
usually the fastest-varying, but occasionally a
different dimension might be dense and slowest
varying.)
For an unpartitioned cube, when you do your initial
load and build, the data will be grouped so that all
time periods for a given combination of PROD and
GEOG are stored contiguously in a LOB - which
means, physically speaking, that they are all on
the same page or small set of pages. When you
add new values to to TIME dimension, however, and
load some new data for those values, those values
will go at the end of the LOB - meaning they will
go on a different page.
This kind of maintenance can cause several problems.
First, it tends to fragment the index that the engine
uses to keep track of where all the data lives, which
slows down everything. Second, it means that when
you delete a bunch of contiguous TIME dimension
values (like, say, everything older than 1 year), you
may wind up having lots of pages that have some
deleted data and some undeleted data. This is bad
because it means that you need just as many page
reads to access your data AFTER you delete a
bunch of data.
On the other hand, if you partition by TIME, you put
limits on how bad these problems can get. If you
partition at the QUARTER level, then you can be sure
that when you want to delete an old quarter, all the
pages belonging to that QUARTER, and all the index
segmentation that they can represent, will be gone.
As to overhead - there's some, but last I heard it
didn't seem to be a big issue for anyone. You probably
want to avoid partitioning on too fine of a level, though,
for several reasons. First, there's some overhead involved
in building each partition. Second, fine-grained partitioning
does disrupt the "locality" of adjacent time periods -
if you partition on week, then the values for one week will
never be stored in the same page as the values for another
week. Finally, I don't think that AWM allows you to
precompute values for dimension positions "above" the
partitioning level. In other words, if you really need to
precompute quarterly totals, you can't partition on month -
you must partition on quarter or year.
Hope this helps.
Dan