Aggregate storage cache warning during buffer commit

Question

h5. Summary
Having followed the documentation to set the ASO storage cache size I still get a warning during buffer load commit that says it should be increased.
h5. Storage Cache Setting
The documentation says:
A 32 MB cache setting supports a database with approximately 2 GB of input-level data. If the input-level data size is greater than 2 GB by some factor, the aggregate storage cache can be increased by the square root of the factor. For example, if the input-level data size is 3 GB (2 GB * 1.5), multiply the aggregate storage cache size of 32 MB by the square root of 1.5, and set the aggregate cache size to the result: 39.04 MB.
My database has 127,643,648k of base data which is 60.8x bigger than 2GB. SQRT of this is 7.8 so I my optimal cache size should be (7.8*32MB) = 250MB. My cache size is in fact 256MB because I have to set it before the data load based on estimates.
h5. Data Load
The initial data load is done in 3 maxl sessions into 3 buffers. The final import output then looks like this:
MAXL> import database "4572_a"."agg" data from load_buffer with buffer_id 1, 2, 3;
OK/INFO - 1270041 - For better performance, increase the size of aggregate storage cache.
OK/INFO - 1270041 - For better performance, increase the size of aggregate storage cache.
OK/INFO - 1270041 - For better performance, increase the size of aggregate storage cache.
OK/INFO - 1003058 - Data load buffer commit elapsed time : [5131.49] seconds.
OK/INFO - 1241113 - Database import completed ['4572_a'.'agg'].
MAXL>
h5. The Question
Can anybody tell me why the final import is recommending increasing the storage cache when it is already slightly larger than the value specified in the documentation?
h5. Versions
Essbase Release 11.1.2 (ESB11.1.2.1.102B147)
Linux version 2.6.32.12-0.7-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP 2010-05-20 11:14:20 +0200 64 bit

Answer 1

My understanding is that storage cache setting calculation you quoted is based on the cache requirements for retrieval. This recommendation has remained unchanged since ASO was first introduced in v7 (?) and was certainly done before the advent of parallel loading.
I think that the ASO cache is used during the combination of the buffers. As a result depending on how ASO works internally you would get this warning unless your buffer was:
1. = to the final load size of the database
2. OR if the cache was only used when data existed for the same "Sparse" combination of dimensions in more than one buffer the required size would be a function of the number of cross buffer combinations required
3. OR if the Cache is needed only when compression dimension member groups cross buffers
By "Sparse" dimension I mean the non-compressed dimensions.
Therefore you might try some experiments. To test case x above:
1. Forget it you will get this message unless you have a cache large enough for the final data set size on disk
2. sort your data so that no dimensional combination exists in more than one buffer - ie sort by all non-compression dimensions then by the compression dimension
3. Often your compression dimension is time based (EVEN THOUGH THIS IS VERY SUB-OPTIMAL). If so you could sort the data by the compression dimension only and break the files so that the first 16 compression members (as seen in the outline) are in buffer 1, the next 16 in buffer 2 and the next in buffer 3
Also if your machine is IO bound (as most are during a load of this size) and your cpu is not - try using os level compression on your input files - it could speed things up greatly.
Finally regarding my comments on time based compression dimension - you should consider building a stored dimension for this along the lines of what I have proposed in some posts on network54 (search for DanP on network54.com/forum/58296 - I would give you a link but it is down now).
OR better yet in the forthcoming book (of which Robb is a co-author) Developing Essbase Applications: Advanced Techniques for Finance and IT Professionals http://www.amazon.com/Developing-Essbase-Applications-Techniques-Professionals/dp/1466553308/ref=sr_1_1?ie=UTF8&qid=1335973291&sr=8-1
I really hope you will try the suggestions above and post your results.