Subcube and consolidation.

Question

Hello Experts,
A new forecast scenario dimension is added. Our process is to extract data, data can be extracted from any existing scenario, and then use that as input into the new scenario and consolidate.
Once the consolidation is complete we clear the data and reload with valid data.
We find that if we do this process before we load the valid data, the consolidation processes (when loading the valid data) goes much quicker than if we were to run the consolidation the very first time with the valid data..
Can you prep a new SubCube (new scenario) in order to make the initial consolidation process run quicker.
*** Example:; If we do not run a preliminary consolidation on a new scenario, and perform a first time consolidation the process takes 35 hours..
If run a pre-consolidation first (invalid data), then consolidation that we do when we have valid data takes 17 hours...
We can do the pre-consolidation on a weekend, and then we are ready to load valid data when the time comes. Since a first time consolidation would take to long time, (customer unhappiness) we have decided to perform a pre-consol.
Any suggestions would be much appreciated.
Thank you,
Charles Babu J

Answer 1

From a SQL Server perspective, but I would think Oracle works similar.....
I disagree that it is necessarily indexes. If your fill factor is too tight, you will get index fragmentation when you add a lot of data. Essentially the index is built out of order and a rebuild will need to happen. Unless you have some sort of automatic index rebuild running at all times (not likely) this is not the immediate issue. I can confirm taht consolidations definitely will fragment index. In regards to fill factor, essentially you are leave gaps in the index so that as you add new entries, they can be put in the right order. If your fill factor is too high, there's no room for new entries and they go to the end. If your fill factor is really low, the indexes will take more space and be slightly inefficient, but they shouldn't fragment as much.
I think perhaps the database is getting expanded ('auto grown') as the data is getting added. When you create a database, you specify a size for the log and data file(s). What happens when the data added to database hits the max size of the database? You can't add any more data and boom. To solve that problem, auto-grow functionality exists. 'Auto-grow' essentially will re-allocate disk space for the database as it needs it. How much does it add? That's up to you. Most system allow you to set auto-growth by MB or % of current size.
If these settings are too small and you are adding a lot of data, the database server will have to continually grow the database which is a time consuming process. Say your database has 65 GB allocated to data, your application is 60GB and one scenario takes up 10GB. If you start copying a scenario to a new (empty) scenario, your database will end up being -5 GB short! Now... lets say you have autogrow set to 1 MB (**DEFAULT in SQL SERVER 2005**), you are in BIG trouble. SQL will autogrow the database thousands of times in 1 MB increments. This will absolutely kill performance.
If that is happening, you will see log entries about auto-growing the database.
With all of that said, proper database file sizing is critical as well as auto-grow settings.
You should be proactively reviewing your data usage and estimating where you will need to be as you implement new features, etc. If you know you are going to add 10 GB of data to your system, you should just manually grow the database to at least that amount.
If you want to rely on auto-grow, you should review the auto-grow settings. the auto-grow amount should be enough that it rarely has to run.