Delay when querying from CUBE_TABLE object, what is it?

Question

Hi Guys,
We are using Oracle OLAP 11.2.0.2.0 with an 11g Cube, 7 Dimensions, Compressed and partitioned by Month.
We have run into a performance issue when implementing OBIEE.
The main issue we have is a delay while drilling on a hierarchy. Users have been waiting 7-12 seconds per drill on a hierarchy, and the query is only returning a few cells of data. We have managed to isolate this to slow performing queries on CUBE_TABLE.
For example, the following query returns one cell of data:
SELECT FINSTMNT_VIEW.BASE, FINSTMNT_VIEW.REPORT_TYPE, FINSTMNT_VIEW.COMPANY, FINSTMNT_VIEW.SCENARIO, FINSTMNT_VIEW.PRODUCT, FINSTMNT_VIEW.ACCOUNT, FINSTMNT_VIEW.SITE, FINSTMNT_VIEW.TIME
FROM "SCHEMA1".FINSTMNT_VIEW FINSTMNT_VIEW
WHERE
FINSTMNT_VIEW.REPORT_TYPE IN ('MTD' )
AND FINSTMNT_VIEW.COMPANY IN ('E01' )
AND FINSTMNT_VIEW.SCENARIO IN ('ACTUAL' )
AND FINSTMNT_VIEW.PRODUCT IN ('PT' )
AND FINSTMNT_VIEW.ACCOUNT IN ('APBIT' )
AND FINSTMNT_VIEW.SITE IN ('C010885' )
AND FINSTMNT_VIEW.TIME IN ('JUN11' ) ;
1 Row selected in 4.524 Seconds
Note: FINSTMNT_VIEW is the automatically generated cube view.
CREATE OR REPLACE FORCE VIEW "SCHEMA1"."FINSTMNT_VIEW" ("BASE","REPORT_TYPE", "COMPANY", "SCENARIO", "PRODUCT", "ACCOUNT", "SITE", "TIME")
AS
SELECT "BASE", "REPORT_TYPE", "COMPANY", "SCENARIO", "PRODUCT", "ACCOUNT", "SITE", "TIME"
FROM TABLE(CUBE_TABLE('"SCHEMA1"."FINSTMNT"') ) ;
If we increase the amount of data returned by adding to the query, it only increased the query time by .4 seconds
SELECT FINSTMNT_VIEW.BASE, FINSTMNT_VIEW.REPORT_TYPE, FINSTMNT_VIEW.COMPANY, FINSTMNT_VIEW.SCENARIO, FINSTMNT_VIEW.PRODUCT, FINSTMNT_VIEW.ACCOUNT, FINSTMNT_VIEW.SITE, FINSTMNT_VIEW.TIME
FROM "SCHEMA1".FINSTMNT_VIEW FINSTMNT_VIEW
WHERE
FINSTMNT_VIEW.REPORT_TYPE IN ('MTD' )
AND FINSTMNT_VIEW.COMPANY IN ('E01' )
AND FINSTMNT_VIEW.SCENARIO IN ('ACTUAL' )
AND FINSTMNT_VIEW.PRODUCT IN ('PT' )
AND FINSTMNT_VIEW.ACCOUNT IN ('APBIT' )
AND FINSTMNT_VIEW.SITE IN ('C010885', 'C010886', 'C010891', 'C010892', 'C010887', 'C010888', 'C010897', 'C010893', 'C010890', 'C010894', 'C010896', 'C010899' )
AND FINSTMNT_VIEW.TIME IN ('JUN11' ) ;
12 rows selected - In 4.977 Seconds
If we increase the data returned even more:
SELECT FINSTMNT_VIEW.BASE, FINSTMNT_VIEW.REPORT_TYPE, FINSTMNT_VIEW.COMPANY, FINSTMNT_VIEW.SCENARIO, FINSTMNT_VIEW.PRODUCT, FINSTMNT_VIEW.ACCOUNT, FINSTMNT_VIEW.SITE, FINSTMNT_VIEW.TIME
FROM "SCHEMA1".FINSTMNT_VIEW FINSTMNT_VIEW
WHERE
FINSTMNT_VIEW.REPORT_TYPE IN ('MTD' )
AND FINSTMNT_VIEW.COMPANY IN ('ET', 'E01', 'E02', 'E03', 'E04' )
AND FINSTMNT_VIEW.SCENARIO IN ('ACTUAL' )
AND FINSTMNT_VIEW.PRODUCT IN ('PT', 'P00' )
AND FINSTMNT_VIEW.ACCOUNT IN ('APBIT' )
AND FINSTMNT_VIEW.SITE IN ('C010885', 'C010886', 'C010891', 'C010892', 'C010887', 'C010888', 'C010897', 'C010893', 'C010890', 'C010894', 'C010896', 'C010899' )
AND FINSTMNT_VIEW.TIME IN ('JUN11', 'JUL11', 'AUG11', 'SEP11', 'OCT11', 'NOV11', 'DEC11', 'JAN12') ;
118 rows selected - In 14.213 Seconds
If we take the time for each query and divide by the number of rows, we can see that querying more data results in a much more efficient query:
Time/Rows returned:
1 Row - 4.524
12 Rows - 0.4147
118 Rows - 0.120449153
It seems like there is an initial delay of approx 4 seconds when querying the CUBE_TABLE object. Using AWM to query the same data using LIMIT and RPR is almost instantaneous...
Can anyone explain what this delay is, and if there is any way to optimise the query?
Could it be the AW getting attached before each query?
Big thanks to anyone that can help!

Answer 1

Thanks Nasar,
I have run a number of queries with logging enabled, the things you mentioned all look good:
Loop Optimization: GDILoopOpt     COMPLETED
Selection filter: FILTER_LIMITS_FAST     7
ROWS_FAILED_FILTER     0
ROWS_RETURNED     1
Predicates: 7 pruned out of 7 predicates
The longest action I have seen in the log is the PAGING operation... but I do not see this on all queries.
Time Total Time OPERATION
2.263     27.864          PAGING     DYN_PAGEPOOL     TRACE     GREW     9926KB to 59577KB
1.825     25.601          PAGING     DYN_PAGEPOOL     TRACE     GREW     8274KB to 49651KB
1.498     23.776          PAGING     DYN_PAGEPOOL     TRACE     GREW     6895KB to 41377KB
1.232     22.278          PAGING     DYN_PAGEPOOL     TRACE     GREW     5747KB to 34482KB
1.17     21.046          PAGING     DYN_PAGEPOOL     TRACE     GREW     4788KB to 28735KB
1.03     19.876          PAGING     DYN_PAGEPOOL     TRACE     GREW     3990KB to 23947KB
2.808     18.846          PAGING     DYN_PAGEPOOL     TRACE     GREW     3325KB to 19957KB
What is strange is that the cube operation log does not account for all of the query time. For example:
SELECT "BASE_LVL" FROM TABLE(CUBE_TABLE('"EXAMPLE"."FINSTMNT"'))
WHERE
"RPT_TYPE" = 'MTD' AND
"ENTITY" = 'ET' AND
"SCENARIO" = 'ACTUAL' AND
"PRODUCT" = 'PT' AND
"GL_ACCOUNT" = 'APBIT' AND
"CENTRE" = 'TOTAL' AND
"TIME" = 'YR09';
This query returns in 6.006 seconds using SQL Developer, if I then take the CUBE_OPERATION_LOG for this query and subtract the start time from the end time, I only get 1.67 seconds. This leaves 4.3 seconds unaccounted for... This is the same with the my other queries, see actual time and logged time below:
Query     Actual     Logged      Variance
S3     6.006     1.67     4.336
L1     18.128     13.776     4.352
S1     4.461     0.203     4.258
L2     4.696     0.39     4.306
S2     5.882     1.575     4.307
Any ideas on what this could be or how I can capture this 4.3 second overhead?
Your help has been greatly appreciated.