Data Load Optimization

Question

Hi,
I have a cube with following dimension information and it requires optimization for data load, its data is cleared and loaded every week from SQL data source using load rule. It loads 35 million records and the load is so slow that only for data load excluding calculation takes 10 hrs. Is it common? Is there any change in the structure I need to make the load faster like changing the Measures to sparse or change the position of dimensions. Also the block size is large, 52920 B thats kind of absurd. I have also the cache settings below so please look at it please give me suggestions on this
MEASURE      Dense     Accounts 245 (No. Of Members)
PERIOD     Dense     Time 27
CALC      Sparse     None      1
SCENARIO     Sparse     None 7
GEO_NM     Sparse     None     50
PRODUCT     Sparse     None 8416
CAMPAIGN     Sparse     None 35
SEGMENT     Sparse     None 32
Cache settings :
Index Cache setting : 1024
Index Cache Current Value : 1024
Data File Cache Setting : 32768
Data file Cache Current Value : 0
Data Cache Setting : 3072
Data Cache Current Value : 3049
I would appreciate any help on this. Thanks!

Answer 1

10 hrs is not acceptable even for that many rows. For my discussion, I'll assume a BSO cube,
There are a few things to consider
First what is the order of the columns in your load rule? Can you post the SQL? IS the sql sorted as it comes in? Optimal for a load would be to have your sparse dimensions first followed by the dense dimensions(preferably having one of the dense dimensiosn as columns instead of rows) For example your periods going across like Jan, Feb, Mar, etc
Second, Do you have parallel data loading turned on? Look in the config for Dlthreadsprepare and DLthreadswrite. My multithreading you can get better throughput
Third, how does the data get loaded? Is there any summation of data before being loaded or do you have the load rule set to addative. doing the summation in SQL would spead things up a lot since each block would only get hit once.
I have also seen network issues cause this as transferring this many rows would be slow ( as KRishna said) and have seen where the number of joins done on the SQL caused massive delays in preparing the data. Out of interest, how long does the actual query take if you are just executing it from a SQL tool.